NeuralOS: An operating system powered by neural networks

I didn't get to do much. Had a hard time clicking on Firefox and then getting to the nav bar and type in "Hackernews". Boy was that wild watching it type. Those definitely weren't letters. Then it tried to translate the page for me into Finish and weirdly the "I'm not a robot" box would appear, disappear, and then I'd see the title of some paper. I never actually made it to the Google results...

It's an interesting project. I'll totally accept "for fun" or "because" but I'm interested in the why. Even if just a very narrow thing, is there any benefits we would get from using a ML based OS? I mean it is definitely cool and that has merit in its own right, but people talk about Neural OSs and I just don't "get it"

Thanks everyone for trying out NeuralOS, and apologies for the frustrating user experience!

I coded up the demo myself and didn't anticipate how disruptive the intermittent warning messages about waiting users would become. The demo is quite resource-intensive: each session currently requires its own H100 GPU, and I'm already using a dispatcher-worker setup with 8 parallel workers. Unfortunately, demand exceeded my setup, causing significant lag and I had to limit sessions to 60 more seconds when others are waiting. Additionally, the underlying diffusion model itself is slow to run, resulting in a frame rate typically below 2 fps, further compounded by network bottlenecks.

As for model capabilities, NeuralOS is indeed quite limited at this point (as acknowledged in my paper abstract). That's why the demo interactions shown in my tweet were minimal (opening Firefox, typing a URL).

Overall, this is meant as a proof-of-concept demonstrating the potential of generative, neural-network-powered GUIs. It's fully open-source, and I hope others can help improve it going forward!

Thanks again for the honest feedback.

I tried to use this but the lag made it impossible to even click on an icon. On top of that, a message that other people were waiting popped up intermittently, pushing the emulation down the page, away from the mouse pointer. I'm not sure what sort of experience you're aiming for, but this probably isn't it.

This is a cool proof-of-concept! It reminds me of https://oasis-model.github.io/. which friends and I had a lot of fun with

This brings personal nostalgia to when I was very young and made an "OS" in PowerPoint using links between slides, animations, and the embedded internet explorer object. Similarly, I'm not sure I see any practical use in this. Still it's a really fascinating conceptual demonstration of networks understanding intent in the complex state-machine that is a graphical user interface.

This reminds me of a recent conversation we had at work where someone suggested that at some point all backend APIs are going to get replaced by a single LLM that'll just do anything (if you ask it nicely enough).

Tried to use this, also found lag made it basically impossible, and felt uncomfortable being reminded that other people might be waiting for me to get on with it.

However, I was able to click on a folder, it opened and looked fairly convincing. Only indicator that something was off - other than lag - was the at the bottom of the file browser, it mentioned how much diskspace was available: the first digit was clearly 6, the second was flickering and blurring between different numbers.

Pretty interesting idea though. What framerates should it run at? I felt I was getting <5fps.

A generative operating system that directly predicts screen images based on mouse and keyboard inputs, powered by an RNN for state modeling and a diffusion model for image generation.

See my tweet for more details: https://x.com/yuntiandeng/status/1944802154314916331

seems similar to this: https://aistudio.google.com/apps/bundled/gemini_os?showPrevi...

although i wasn't able to really use it due to lag

You can visit NeuralOS inside NeuralOS!

Who thought this was a good idea?