OmniParser for Pure Vision Based GUI Agent
34 comments
·October 25, 2024deergomoo
franga2000
This is not a computer problem, it's a human one. It's not that we don't have APIs and hooks because they're so difficult to implement - we don't have them because software producers don't want or care for us us to have them.
Enabling automation will never be zero effort and anything more than zero effort for something with such a low ROI is a no-go by default. But increasingly, automation is actually seen as a danger to their business models and companies sometimes even go out of their way to prevent it.
Looking at the screen the same way a user does is the only way to win.
Fulgen
Focusing on the GUI applications: There have been a few GUI automation solutions over the years - since the post's software is from MS, I'll take UI Automation as an example. Works well with Win32 controls, not sure how well it works with the XAML-based toolkits.
But not all software is written with those UI frameworks. Some use different widget frameworks, some immediate GUIs, others just render a webpage and either use HTML or fully render the controls themselves. And without everybody using the same standard, the only standard we have for parsing their output is the pixels they render to.
mountainriver
Computer based agents have no limits, that’s the advantage. Sure a proper automation hook is better if its available, but a lot of the time it isn’t, either due to lack of resources or monopolistic behavior
Smaug123
To a considerable extent, we are stuck in the world we live in; but I am reminded of a quote by Guillaume Allais:
> My entire job seems to be repeating variations of "never start by forgetting the user's stated intent only to then attempt to guess it".
trq_
This is awesome, can't wait for evals against Claude Computer Use!
amelius
Can we first test this with basic sysadmin work in a simple shell?
Can't wait to replace "apt get install" by "gpt get install" and then have it solve all the dependency errors by itself.
ErikBjare
This had been possible for a year already. My project gptme does it just fine (like many other tools), especially now with Claude 3.5.
amelius
I know that it exists. I was just hoping we can make such interactions (practically) bug-free before we move on to the next big thing.
anonym29
Threat actors can't wait for you to start doing this either.
asdev
how can you write metrics against something that's non deterministic?
amelius
Can it detect ads and mask them out?
bee_rider
If these sorts of tools kill the ad business, it would be so incredibly cool, and justify Nvidia’s half-of-the-economy-or-whatever market cap.
amelius
Let's hope so! But now that I'm thinking about it more: nvidia might go into the advertisement business themselves :(
dartos
Only like 6%
dymk
I’m reminded of Permutation City where your personal AI intercepts ads sent to you, but ad companies of course have their own AI for tricking your AI, so of course you have a countermeasure AI to intercept that, and so on and so forth
jauntywundrkind
I have a little bit of a vice of enjoying some "idle" games. I have intended to do some very basic manual screen carving & ocr & computer vision to try to "read" my state in these games, & have multi-actor "play" models for them, just for fun really & to decrease time sunk gaming (by spending significant time coding/learning).
This certainly seems like it has a lot of promise to make that much much much easier. Game UI's are less uniform so maybe this might be harder or not easily be applicable, but hopefully
_adamb
As someone who has done this to many games over a few decades, I can definitively say: 100% of the time, it ruins the fun of the game.
I can't say exactly why. Maybe you feel like you haven't earned it. Maybe it's the idle nature of farming that we really enjoy...
fragmede
Depends what you consider fun, and how far you take it. Some people enjoy programming more than repetitive clicking in a GUI. For a clicker game, writing a bot lets you iterate on strategies easier - is it faster to get to level 2 if I buy the upgrade for A or B first? For Trackmania, it lets you get a world record and a YouTube video with 14M views.
jauntywundrkind
Yeah. I appreciate the warning & enjoy the personal tail, but it's just that guys story & it's being projected as an absolute.
If I don't enjoy the experience anymore that's fine with me too. I think I'd still feel a sense of accomplishment, feel like I'd advanced as a human and mastered my environment and machines for diving in here.
I don't feel the agency I want to have. These games make me want to extend myself, my agency. Playing them manually offers some very low grade enjoyment but that sense of missing out gnaws at me, and I'm not at all dissuaded by parent trying to ward me off, and if I do end up winning so hard I don't care anymore, me right now would regard that as a victory condition & rief from this pressure I feel about ineffectively plodding through as I do now.
nmstoker
You might want to look at Serpent AI: granted the repo is now in an archived state, but it did similar things to those you mention.
patrickhogan1
Since this is a research paper with promising ideas but non-functional code, what are people using as the best-in-class agents for computer automation? For example:
1. Claude for computer use
2. Various startup offerings—if you have recommendations, please list them
3. Established tools like Playwright, Selenium, and WebDriver, combined with screenshots and LLM-based guidance
What tools or approaches are actually working for building useful automation solutions?
nmstoker
Are you sure about the non-working code point?
I've yet to try it but my understanding is the repo here has got working code along with installation instructions:
nmstoker
I confirm it works: I got the gradio demo working locally and it's pretty reasonable.
Slight rough edges (to be expected) and you do need to read the README with attention but it's all par for the course. I had to install einops which wasn't in the requirements.txt and even though I had downloaded the HF models they released, it still needed to pull in another model when I first ran the demo.
patrickhogan1
Thanks for the tip, will try again.
tomatohs
our agent is available via NPM: http://testdriver.ai
akshayKMR
Does it also tell the coordinates (x,y) of the annotated box w.r.t. the screenshot dimensions?
anonzzzies
Computer Use, Agent.exe and so on, but nothing actually is useful yet. It's all very terrible. And then to think we had perfection already (and Claude is good at it); emacs... No need for any of this; everything can be scripted.
s3tt3mbr1n1
Has anyone gotten this to work?
Copying the repo and downloading the models through HuggingFace or manually does not seem to work, you get errors indicating missing files.
suriya-ganesh
I tried as well. Seems like it is a proprietary model.
bbor
:( I was so, so excited to try this when I found it yesterday, it has like 3 star emojis in my list of models. I’ll post here if I get it working tomorrow, I guess. I doubt they’d release a model on HF without intending to make it useable
EDIT; surely it’s just broken, the repo does include .safetensor weights. Maybe the problem is the “suspicious”-flagged PyTorch weight for “icon detection”, whatever that means?
dartos
FOSS-washing?
nmstoker
This literally just published, IMHO it's a little premature to be accusing them of that at such an early stage.
More likely they just slipped up with getting everything uploaded properly - it's easily done, and luckily easily corrected, so we'll likely see issues get resolved fairly swiftly.
One thing I have found very odd about the current wave of AI tools is that there seems to be an unspoken element of giving up and admitting failure in other areas of computing.
Programming copilots are often sold on how they can automate drudgery and boilerplate, which implies we are incapable of or uninterested in designing programming languages, tools, and patterns which do not require boilerplate or drudgery.
Teaching models to use traditional GUI apps implies we have given up on or are not even bothering to create proper hooks for an automation system to utilise.
Something about it feels wrong to me, because it bakes existing inefficiencies into the system. Can we really not solve the inefficiencies instead of pouring unfathomable amounts of compute into working around them?