RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning (2023)
58 comments
·February 27, 2025pama
djtango
Tangential but your comment has just given me a penny drop moment that I can isolate each degree of freedom (finger, wrist, elbow, shoulder) and still play a key to sound a note.
For context I have been working on playing multiple lines of music with right hand lately (Chopin etudes, much struggle...) and saw a video from a professional recommending playing the main melody "with the arm". This has helped a lot with the visualisation and now I can really have a cleaner technique playing the accompaniment above the escapement (using nice flat fingers) and the melody with a much richer tone (using the arm typically for the pinky).
I sort of understood in an intuitive nonverbal sense what I am doing but after your comment, the mechanics now makes a lot more sense of what I'm actually doing - thanks!
pama
Glad it helps. Claudio Arrau thought of the fingers as “dead” with all the piano playing happening using larger muscles. It is a struggle to internalise this concept in some Chopin etudes (op 10, no 2 especially) but it almost immediately helps in others (op 25, no 1).
bondarchuk
Interesting, I've read similar things about calligraphy/writing (from the elbow!), hadn't connected it to piano playing.
cbtones
Yes, that's where this model is massively missing, an accomplished pianist plays with the whole arm, the elbow is crucial, it needs to stay very relaxed in order to accommodate the peculiarities of the human hand, the ring fingers and little fingers don't have as many tendons connecting and having a little sway in the elbows is crucial to making those fingers deliver the weight of the arm properly to the keybed. This model appears to ignore rotation of the wrist, and easing the elbows, where at a bare minimum it needs to start at the shoulder, and I'd argue the whole body.
djtango
Yeah but as gp pointed out that was the style du jour of baroque and early classical as the plucking keyboards were mostly played with a finger technique
vunderba
This. In fact isolating and attempting to play purely from the fingers alone is a great way to develop RSI or any number of soft tissue injuries.
There's entire schools of thought around proper ergonomics with respect to piano playing - I took lessons in the Taubman technique which (very simplistically) tries to encourage movement farther upstream on the body.
https://www.ednagolandsky.com/the-taubman-approach-basic-pri...
kbouck
Any way we can get the Westworld intro song into the demo list?
ericye16
This was the basis of a project I did for my deep reinforcement learning class!
https://ericye16.com/stanford-cs224r
We were able to make some improvements by tuning how the reward is distributed and also by first pretraining the agent on scales before fine-tuning them on the final pieces.
Thanks to Kevin Zakka for helping us get started with the RL environment!
plaguuuuuu
did you guys ever try having the agents play the song slower at first?
ericye16
We definitely tried extending the lookahead, but I don't think we tried having a curriculum-style thing where we gradually increased the speed of the song. Great idea though!
kittikitti
> UC Berkeley, Google DeepMind, Stanford University
I've personally experienced how research around this time was being shut down because of AI doomerism. People were getting laid off because of it. It's clear to me that these institutions actively spread AI doomerism so that they have full control over it. They actively called for a stop in AI research so that their personal labs can leap forward ahead. It was a little too on the nose for Big Tech, but they don't understand nuance.
criddell
Is this actually buildable? I'd like to hear it on a real piano because (IMHO) the one in the videos sounds bad.
I'd like to also hear how loud the mechanical noise of the machine playing the piano would be. Does the left hand work harder with the heavier keys? What would the hands be mounted to?
dheera
> We use the Musical Instrument Digital Interface (MIDI) standard to represent a musical piece as a sequence of time-stamped messages corresponding to "note-on" or "note-off" events. A message carries additional pieces of information such as the pitch of a note and its velocity.
I play the piano and I think MIDI does not have enough parameters to describe an acoustic piano. It's not just a single strike velocity that determines each sound. Where you play within the double escapement, how far down you hit the key, they all change the sound a bit.
That said though there are better MIDI synthesizers, e.g. https://www.modartt.com/ They still don't match an acoustic. I can tell the difference quite easily. I think a neural TTS retrained on piano data could do better.
cloudbonsai
> Is this actually buildable
Yes. The robot used in this study is Dexterous Hand from Shadow Robot. It's a real product, and costs around $200-300k (for a single hand).
Controlling a real robot using a RL policy (model) trained in a simulation environment is also doable. It's called "Sim2Real" and has been widely experimented in the last decade (with mixed success, though).
criddell
Thanks for that information.
There's a video that shows the hand moving and it doesn't look like it would be fast enough for anything but relatively slow songs:
knowaveragejoe
This is more recent, and arguably going after something different(fingertip sensation), but I'm sure has overlap as well:
j_bum
What do you mean “heavier keys”?
Each key (on a good piano) is weighted to have the same force requirements to be played. Of course varying forces can be used to achieve different decibels, but the curve is the same across the piano.
ano-ther
Probably this:
> Every single key on a grand piano keyboard is weighted differently. This is because the strings for each note are slightly thinner and shorter in the treble register, becoming thicker and longer towards the bass register. As a result, there is greater resistance when playing low notes than when playing high notes. In other words, a heavier touch is required in the left hand and a lighter touch in the right hand. To emulate this in a digital piano, the keys are often individually weighted, with the lower keys heavier than the higher ones — something that’s called graded weighting.
https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-wei...
cbtones
It's not the strings though, the hammers are smaller as you go to the right in the higher register on the piano. The strings are slightly smaller in diameter, but remarkably shorter in length.
rossant
Nice work! And great interactive 3d application. My 6yo had a lot of fun annoying the robot while it's playing by forcefully moving its hands around.
null
beardyw
All the right notes but not necessarily at the right time.
kzrdude
Which is exactly the wrong way around from what's expected for a pianist :)
Tycho
Has anyone tried to get a robot to do oil paintings?
adamheins
This company: https://www.acrylicrobotics.ca/
fedeb95
waiting for Robot Devil.
davidanekstein
Funny enough I’m listening to Rhapsody in Blue while browsing HN. I’d like to see it do that for 17 minutes.
marbro
How does this research help us win naval battles? Should Elon and his DOGE friends cut it?
This was a nice demonstration of letting these robotic hands learn to play a keyboard. The technique is limited due to constraints of the hands, and is closer to parts of the technique of early keyboard instruments like an organ or a harpsichord, rather than that of a modern grand piano, which requires a lot more control of the body core, shoulders, elbows, arms, and wrists with the fingers doing as minimal a motion as possible. I suppose a similar algorithm learning to play on a grand piano using a full humanoid body could learn a technique that would be exciting to analyze.