I put sheet music into smart glasses [video]
48 comments
·May 3, 2025gavinray
I am so unbelievably excited for consumer-grade, useful AR.
There was a lot of hype around VR, but for the last 10 years I've been following progress on AR glasses.
The thing about AR is that it has the ability to enhance everything in your daily life, versus VR which is meant to be a separate experience.
Both Meta and Samsung are due to put out consumer AR glasses later this year and I think this might be the first wave of useful, daily-wear glasses we'll see.
Is there anyone who works in the AR space that could comment more?
caydenpiercehax
100% agree.
I've been building smart glasses for over 7 years. First 6 years were in academia because the tech wasn't ready yet, because they were too heavy and battery didn't last long enough.
But in the last year, all-day battery smart glasses have become lightweight enough to be worn all day (see Even Realities G1, Vuzix Z100, etc.).
I believe smart glasses are having their iPhone moment in 2025 + 2026.
We make the smart glasses OS that Kevin used in the video to make this smart glasses app: AugmentOS.org
apples_oranges
Unlike you i think the potential is rather in business and work. Useful info when needed.
null
gavinray
Just bought a pair of the Mentra Mach1 glasses, let's see how they pan out.
febed
Any workaround for those needing prescription glasses?
alex1115alex
You can get prescription inserts from Vuzix, but they're pretty bad. If you need a prescription, and want to run AugmentOS, your best bet is to buy the Even Realities G1 instead.
dang
This project is cool so we're hoping to arrange with Kevin to do a Show HN about it, so stay, er, tuned!
null
swyx
wow that's a special honor. is there a way to search up the "specially invited" Show HNs?
alex1115alex
One of our users documented projecting sheet music onto his smart glasses's display (with a HUD). He did a great job documenting the limitations of 2025's tech, but it gives a great look into what's going to be feasible next year.
Awesome job Kevin!
vunderba
Nice work. From the Github:
> This allows the pianist to not have to turn pages, and more importantly, allows them to see the music and their hands at the same time, which is an unavoidable problem with traditional sheet music.
I could definitely see this being beneficial for beginners. When I lived in a dormitory during uni I often played familiar pieces from memory pretty late on a digital piano (with headphones) in extremely dim lighting so as not to disturb my roommate.
At some point I just stopped having to look down at the keyboard. I play a lot of stride piano as well and that probably conditioned me to just have a sort of musical proprioception for the instrument. And of course, there's numerous examples of unbelievable blind pianists - Stevie Wonder, Ray Charles, Art Tatum, etc.
kevinlinxc
For me, it's still easy to mess up for complex sections if I'm not peeking at the keyboard every so often. Its true that muscle memory takes over after you reach a level of familiarity but not quite to the extent of biking
dylan604
How many people still look at the keyboard when typing? At some point, you just don't need to look at whatever it is your doing. Also, at some point, you memorize the music if it's something you are playing enough. What level of pianist is reading sheet music and looking at their hands at the same time.
bambax
> I just stopped having to look down at the keyboard
Maybe the next step is an app for people who don't read sheet music; it would light up the keys on the keyboard that you need to press, when you look at it...?
Same for guitar, highlight where to put your fingers on a fret for each chord.
schwartzworld
Various products have done this over the years. Forget app, actual keyboards and guitars that light up. It’s not a good way to learn.
bambax
It's true that it's not a good way to learn, but it's fun for a little while.
That's why an app on glasses for this is better than a complete alternative instrument; the app is much cheaper and should work with any real instrument.
atoav
I wanted to say, after a certain level looking at your fingers when playing an instrument becomes the equivalent of looking at your legs when riding the bicycle.
When I start to think too much about what my fingers are doing I will play worse. For if I want to practise a particular part where I get the fingering wrong, sure, but when you play it for real, looking is counterproductive.
Something like this could be great for beginners tho. But simular to automatic guitar tuners I am not sure if you should get into the habit of this technology being around.
yusina
Counter point: people get too hung up on staring at the sheet. The sheet is just a tool to help you remember what you intend to play. The goal should always be to not need it anymore, and while using the sheet, it's like using a crutch.
The "looking at your fingers" challenge then becomes that you start to play "by eye" instead of "by ear" (or "by feel") which I find is very hard to overcome. Especially when you are improvising.
Though in a sense "by sheet" is just as bad.
schwartzworld
> The goal should always be to not need it anymore, and while using the sheet, it's like using a crutch.
Uh, why? Lots of pros use sheet music, especially for complex pieces. I’ve never heard of an orchestra conductor insisting everyone be off-book.
It’s one thing to memorize pop songs or whatever, but nobody is out there shaming people for not memorizing Rachmaninov
schobi
This is really awesome!
I was surprised about using dilation. I would have expected music21 to support rendering to a certain resolution/dpi setting directly and avoid rescaling the images. But from the music21 documentation this is not obvious how to do it. Rendering music to a low dpi screen nicely (pixel perfect) could circumvent some of the hardware limitations in the mid term.
bambax
This is a really cool project.
From the end of the video:
> I had the bars of music auto-sending at a preset interval. The pedals, instead of flipping bars, temporarily pause the flipping or speed it up, in case I'm desynced from the glasses.
That's how teleprompter apps work. Of course the difference is that when speaking you can pause a little if you get desynced, while with music you're like "on a train" and if you pause, it shows. But having an interval is not shocking.
Maybe the problem with this is that typically, sheet music resolution is not constant -- if there are many short notes it will result in a larger space on the page (a larger bitmap) than if there are few long notes.
So maybe an approach is to send a fixed number of bars, regardless of their actual size, so that the interval can have a constant relation with the tempo of the piece?
> My dream smart glasses would just listen to the performance and automatically flip bars
Couldn't the phone do that? The phone is already the part doing most of the work.
hougaard
Fun, did that 8 years ago with the original Hololens https://www.youtube.com/watch?v=E6cBX4t2kX0
Abishek_Muthian
Congratulations for the project and for winning the hackathon, nicely done!
I am looking for hackable smart glasses with camera which doesn't rely upon any proprietary service to work, Mentra seems to have a camera version but this video seems to suggest that we need to use their service all the time?
caydenpiercehax
Mentra Live is indeed a camera + speakers + microphone pair of lightweight smart glasses - that you can build for.
Mentra Live runs AugmentOS, so you can control all the I/O (camera, speakers, mic) in your own app with the AugmentOS SDK.
Regarding use the backend service all the time: Most apps/developers connect through official AugmentOS.org servers and focus on building their apps, but you can self-host your own backend if you want.
simonjgreen
Super cool PoC. Also an advertisement for the value of local processing over cloud.
hhyndman
What a great idea. I am a musician and use an iPad for my scores. It would be wonderful to replace my glasses with a pair that can display the music.
I noticed the iron ring on your pinkie -- Canadian engineer?
kevinlinxc
Correct!
swyx
TIL about their glasses https://mentra.glass/
great video editing, OP. loved the playthru at the end with the text. you have real talent here, keep giong
kevinlinxc
Thank you, the typewriter effect is surprisingly hard, requires a pretty complicated expression in after effects
seabass
Really awesome project! I’m reminded of something else in the AR/music space from a few years back. Someone made a VR passthrough app to project synthesia-style keyboard overlays onto your actual piano keys. Always cool to see what new hardware can enable. Congrats!
https://github.com/kevinlinxc/AugmentedChords