RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning (2023)
19 comments
·February 27, 2025pama
djtango
Tangential but your comment has just given me a penny drop moment that I can isolate each degree of freedom (finger, wrist, elbow, shoulder) and still play a key to sound a note.
For context I have been working on playing multiple lines of music with right hand lately (Chopin etudes, much struggle...) and saw a video from a professional recommending playing the main melody "with the arm". This has helped a lot with the visualisation and now I can really have a cleaner technique playing the accompaniment above the escapement (using nice flat fingers) and the melody with a much richer tone (using the arm typically for the pinky).
I sort of understood in an intuitive nonverbal sense what I am doing but after your comment, the mechanics now makes a lot more sense of what I'm actually doing - thanks!
pama
Glad it helps. Claudio Arrau thought of the fingers as “dead” with all the piano playing happening using larger muscles. It is a struggle to internalise this concept in some Chopin etudes (op 10, no 2 especially) but it almost immediately helps in others (op 25, no 1).
kbouck
Any way we can get the Westworld intro song into the demo list?
criddell
Is this actually buildable? I'd like to hear it on a real piano because (IMHO) the one in the videos sounds bad.
I'd like to also hear how loud the mechanical noise of the machine playing the piano would be. Does the left hand work harder with the heavier keys? What would the hands be mounted to?
j_bum
What do you mean “heavier keys”?
Each key (on a good piano) is weighted to have the same force requirements to be played. Of course varying forces can be used to achieve different decibels, but the curve is the same across the piano.
ano-ther
Probably this:
> Every single key on a grand piano keyboard is weighted differently. This is because the strings for each note are slightly thinner and shorter in the treble register, becoming thicker and longer towards the bass register. As a result, there is greater resistance when playing low notes than when playing high notes. In other words, a heavier touch is required in the left hand and a lighter touch in the right hand. To emulate this in a digital piano, the keys are often individually weighted, with the lower keys heavier than the higher ones — something that’s called graded weighting.
https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-wei...
rossant
Nice work! And great interactive 3d application. My 6yo had a lot of fun annoying the robot while it's playing by forcefully moving its hands around.
beardyw
All the right notes but not necessarily at the right time.
kzrdude
Which is exactly the wrong way around from what's expected for a pianist :)
davidanekstein
Funny enough I’m listening to Rhapsody in Blue while browsing HN. I’d like to see it do that for 17 minutes.
linwangg
This's awesome!!!
fedeb95
waiting for Robot Devil.
amelius
I mean, keys are always in the place where you expect them to be, so what is the challenge here? (Yes, too lazy to read the article).
cloudbonsai
> what is the challenge here?
If you know the precise state of each motor in a robot hand, you can compute the absolute position of its fingertip, right?
To make a robot play piano, you need to solve the reverse problem of it i.e. to compute the motor states from the target robot position. This problem is called "Inverse Kinematics" and not fully resolved to this day.
Reinforcement Learning (RL) people claim that, if you let a robot move randomly, and give it apporopriate rewards depending on the resulting state, eventually it learns to solve the problem by itself (just like a human baby eventually learns how to walk through trial & error).
Now, you are looking at the state of the art of RL in 2023.
smokel
The idea is that the motor control is pretty hard to program, and by using reinforcement learning, one only has to tell the computer whether it hits the right keys (reward).
The reinforcement learning algorithm takes in the state of the fingers and piano keys, and outputs the required motor actions to perform well.
Also, as you are too lazy to read an article but still feel entitled to comment on it, I herby give you -1 reward. Let's see if that helps to improve your character.
rossant
I think the robot learnt to play by itself, right?
PatronBernard
The thing needs a teacher, terrible technique! /s
AIFounder
[dead]
This was a nice demonstration of letting these robotic hands learn to play a keyboard. The technique is limited due to constraints of the hands, and is closer to parts of the technique of early keyboard instruments like an organ or a harpsichord, rather than that of a modern grand piano, which requires a lot more control of the body core, shoulders, elbows, arms, and wrists with the fingers doing as minimal a motion as possible. I suppose a similar algorithm learning to play on a grand piano using a full humanoid body could learn a technique that would be exciting to analyze.