Single RGB camera turns your palm into a keyboard for mixed reality interaction
6 comments
·May 19, 2025rspoerri
CGamesPlay
That isn't every real application. There are lots of use cases where you need an iPod-shuffle-level interface; and this would appear to be just fine for those.
yaris
When looking at this I remembered two projects I saw several years ago: one was using a wearable camera and colored markers on fingers plus some computing to recognize gestures, e.g. making a square with fingers would command the camera to do a "screenshot"; the other one was using giro-sensors put on fingers to recognize finger and hand movements. Both looked interesting, but I have not heard about any of them since then.
harshitaneja
I think you are referring to Pranav Mistry's Sixth Sense(https://www.pranavmistry.com/archived/projects/sixthsense/). It was part of his research at MIT media labs. I don't think he took it any further. Maybe got incorporated in some ways at Samsung where he headed research.
GoblinSlayer
Why not go Accel World way and just show AR keyboard hanging before you?
alexisread
Looks nice, but it should really say single RGB camera AND a boatload of processing power. TLDR the camera feeds 1280x720 images at 120fps to a PC (with graphics card). While you could reduce this to an on-chip processing block, it will also have to deal with people looking away halfway through typing etc. which adds to the processing overhead, and requires integration with the headset (6 axis sensors etc.).
IRL this has been somewhat tried with the Humane AI pin, with gestures, less than instantaneous response is noticeable, so latency has to be paramount.
Don't mean to rain on the parade as it's a nice piece of work, but comparing to say chording (low cpu requirements, better accuracy, faster, but you have to learn how to chord) shows the different tradeoffs.
> a touch recognition accuracy of 97.0%, which makes Palmpad quite reliable.
But completely unusable for any real application where you'd do a lot of typing.