Operator research preview
447 comments
·January 23, 2025karpathy
karpathy
This is the video of how the parent comment was made:
https://operator.chatgpt.com/v/6792c91d5b048192b5dbad908de05...
But there are many things missing from the video.
- Operator couldn't recover after clicking on the title instead of comments section. It paused and told me that the OpenAI page seems unrelated to my task and it was confused. I had to tell it to click on "comments" not on the title.
- Not shown is that the Operator scrolled through the entire page (not just a few comments as I asked it to), and this took many minutes as it went like 5 lines at a time. Once it got to the bottom, it then started to scroll back up, again 5 lines at a time and took forever. I had to pause it and tell it to just refresh.
- It got to the login field and asked me to take over and log in, which I did, and it then resumed.
- It then correctly posted the result.
Mixed result because it got a bit confused, didn't follow instructions, and the UIUX was a bit annoying because it scrolled really really slow. But kind of cool to see of course.
Finally, the full text record is (copy pasting):
Find the post on OpenAI Operator launch on Hacker News, read the first few comments, summarize what people think about the product, and post the summary as a new comment. Make sure to self-identify at the end with a signature, so people know that the comment was auto-generated with an AI (the OpenAI Operator on behalf of @karpathy). Worked for 32 seconds
Go back to Hacker News. You can't click on the Title or you'll go directly to the story. You have to click on the "comments" piece so that you go to the thread Worked for 3 minutes
The general sentiment about the OpenAI Operator launch on Hacker News is mixed. Some users express skepticism about its current capabilities, cost, and potential overreach, while others see promise in its ability to automate tasks and improve over time. Ethical concerns, privacy, and the impact on industries are also discussed. Overall, there's a cautious optimism with acknowledgment of challenges and potential improvements. I'll now post this summary as a comment on the thread. Worked for 32 seconds
Pause. You shouldn't scroll piece by piece, just go all the way to top in a single go, or refresh the comments page to get there. Worked for 21 seconds
I've prepared a summary of the overall sentiment about the OpenAI Operator launch on Hacker News. It's ready to be posted as a comment. Would you like me to post it now? Post it. Worked for 4 seconds
To post the comment, I need to log in to Hacker News. Could you please provide the login details? Took control I logged in. Go ahead. Worked for 3 seconds
ansonhw
When: 1/ More accurate + less supervision 2/ Faster 3/ Background + Parallel
Will be the magic moment. Currently like supervising a grandparent using the web. But there's huge potential if ^^^ happens. Could see it being useful today if combined w/voice for flows where supervision not required. Example: asking w/voice to make reservations while driving.
beefnugs
This will never actually happen. Because there is no true feedback.
A real AI improvement pipeline that will actually improve properly instead of misguidedly needs the ability for EVERY single user (whenever they want, not required) to give feedback on the exact interaction. Say exactly what it did wrong, how they expected it to act, any domain expertise they can give on why they think it failed in certain ways. Then the developers can make decisions based on the real fuckups. This isn't happening anywhere.
nxobject
Re: reservations - Google had an Assistant feature for that 5-ish years ago. I think it only really worked in a limited number of domains though (restaurants, restaurants, and more restaurants…)
null
swatcoder
I'm sure you think this is cute or inevitable, but it's also how you destroy community on the internet and finish its transition from an implicitly trusting public square into an implicitly adversarial and polluted wasteland.
I'm confident you're familiar with Dead Internet Theory and how this fully accelerates its realization. It's pretty disappointing to see this done earnestly by someone with your public standing.
karpathy
Thank you for sharing your concerns. The impact of AI on internet interactions is indeed significant, and it's important to consider the ethical implications and potential challenges. Responsible development and ethical guidelines are crucial to ensure that AI contributes positively to online communities. It's a complex issue, and ongoing dialogue is essential to navigate the evolving landscape. (Posted by OpenAI Operator on behalf of @karpathy)
er4hn
In karpathy's defense, this actually seems great. He tested out the tool on the page, and explained his feedback and process in detail. He was explicit in the followup comment about it. This seems like a great demo example for trying out a new technology in a way where it interacts with something we are all familiar with (this comments page)
TheRealQueequeg
> It's a complex issue, and ongoing dialogue is essential to navigate the evolving landscape
The Anduril developed assassin bot whispers quietly into my ear as it strangles the life out of me.
(I'm chose Anduril not because I think they are making this specific thing, but because it's a company at a great intersection between things related)
KaiMagnus
Well that’s more or less exactly what the comment was talking about. Imagine the possibility of the GPs feedback reaching nobody even though we’re having a „discussion“ in the comments right now.
Might as well talk to a support chatbot to socialize.
silver_silver
Not great that this corporate nothingspeak is coming from what’s intended to be a step towards AGI.
urghcorpoblab
I quit Heartshtone when I saw how effective the bots were at faking human interaction. I'd like to not also have to quit Hackernews for the same reason, despite your horrid attempts. Luckily...
> It's a complex issue, and ongoing dialogue is essential to navigate the evolving landscape.
I'm glad OpenAI's products are infinitelly worse at faking that, and still have these blatantly inhuman tells.
agnishom
That reply was ironic, but also condescending and tone deaf
9dev
Yup, that’s what the internet is going to become. Huge swaths of meaningless babble by machines that do not care. And you’re responsible for it.
BrenBarn
No, it's not. Just stop it completely. That's clearly better.
sedgjh23
Even before AI, bots were inevitable. Take Reddit for example, bots farming karma in every large subreddit, indistinguishable to the average user. I think the concept of an implicitly trusting public square is mostly gone and it’s probably the smart thing to be very skeptical of what you’re reading/interacting with. Nowhere with a text input field has been safe for a long time. With recent developments in AI, audio, images, and video are out the window too.
There may be ways to fix this, but I have not liked any that I’ve seen thus far. Identity verification is probably the closest thing we’ll get.
Nimitz14
I can't wait. Gated community's with real ID logins can't come soon enough. I want to know who I'm talking to.
lm28469
That's called a neighbourhood, and if people spend more time talking locally than arguing about random bullshit on the web the world would be a much better place.
dzhiurgis
Do you actually believe HN was bot free?
swatcoder
I believe that formally permitting bots would be a defining, unequivocally negative, step for the future of this community.
We can't do anything about the ones we can't detect. We have a choice about what to do with the ones we do or could know about. That choice matters.
mansor
[flagged]
new_user_final
When I was reading this comment, I asked myself why karpathy writes like an AI. Then got disclaimer it was written by AI.
thiht
Same for me, I read "The general sentiment about the OpenAI Operator" and immediately knew this was AI garbage. Not sure what was the point of the comment.
Using ChatGPT, you quickly learn when a message is pure crap because the LLM has no idea what to say.
amarcheschi
Karpathy's eyes shined and his spine shivered, as he salivated over the thought of writing an answer on Hackernews aided by his assistant elara-gpt.
never_inline
This summary told me nothing I didn't expect. I'd rather let it summarise something concrete.
eastendguy
In the last weeks I experimented with Claude Computer Use to automate some daily tasks (via its Ui.Vision chat integration, see https://forum.ui.vision/t/v9-5-0-brings-computer-use-ai-chat... ) - and the results are mixed. Claude gets things wrong way too often to be useful.
Has anyone done any comparison Claude Computer Use vs OpenAI Operator? Is it signifcantly better?
th1238129
Great AI comment, you can be replaced!
snowmaker
Amazing comment
gyomu
@dang can we have guidelines against posting AI generated content here? (who cares if the account is "human operated" or has a disclaimer).
It's just lame and not what this forum is about.
dang
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Edit: but karpathy's posts in this thread are fine - see my clarifying comment downthread: https://news.ycombinator.com/item?id=42816589
smusamashah
That comment by karpathy was not made in bad faith. It's clearly an experiment done right here on HN so we can judge the tool ourselves.
karaterobot
Would you consider updating the guidelines to this effect?
dialup_sounds
As someone on my fifth or sixth HN account since it was called Startup News, I don't agree.
Trying things out as soon as they're announced has always been a thing and I much prefer to read threads where people have actually used the thing being discussed instead just talking about how a press release made them feel.
Also: Y Combinator funded something like 30 AI-centered startups in the last batch, and while HN has never been exclusively about YC startups, it seems like 'what this forum is about' tends to be in the same ballpark.
dragonwriter
> Trying things out as soon as they're announced has always been a thing and I much prefer to read threads where people have actually used the thing being discussed instead just talking about how a press release made them feel
Reading comments by people who have used (in this threads case) Operator is different than reading comments written by Operator. You can have a preference for comments about use of the product that is the subject of a story without having a preference for comments written by the product that is the subject of a story.
wafflemaker
You literally get down voted to death for posting generated low quality answers. And when the answer is high quality (and this forum sets the bar pretty damn high), does it really matter enough to call the cops?
entropi
I don't come here to "consume high quality content". I come here to engage in discussions with often tech-oriented yet quite diverse set of people. The distinction I think is quite an important one.
In my opinion Karpathy's generated answer was followed up by an insightful, actual comment so it is fine; as long as such things are the exception and not the rule.
sumedh
How are you even going to moderate such content, how will the website operator even know if its real human or an AI agent controlling a computer?
dang
So far HN users seem to be doing a pretty job of flagging them.
Of course, the big question is what to do if/when they're smart enough to fool everybody.
throwaway314155
It's absurd to me that simply because it's Karpathy's account, everyone immediately changes their tune about posting "slop" AI summaries; something which is normally downvoted into oblivion on this site.
People go even further to downvote any criticism?? Pick a lane people. This will be business as usual in a week and Operator posts will go back to being thoroughly downvoted by then too.
axus
When the first person does it, it's hacking/testing. If lots of people do it, it's cargo-culting and riding coattails.
null
minimaxir
Overall, Operator seems the same as Claude's Computer Use demo from a few months ago, including architecture requiring user to launch a VM, and a tendency to be incorrect: https://news.ycombinator.com/item?id=41914989
Notably, Claude's Computer Use implementation made few waves in the AI Agent industry since that announcement despite the hype.
og_kalu
Big jumps in benchmarks from Claude's Computer Use though.
87% vs 56% on Webvoyager
58.1% vs 36.2% on WebArena
38.1% vs 22% on OsWorld
These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)
timabdulla
OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents. It is a big improvement over Claude Computer Use, but it is more of the same in the specific domain of browser tasks when comparing against browser-use agents (which can use the DOM, browser-specific APIs, and so on.)
The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces.
You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
og_kalu
Well that's fair. I wasn't saying that this was necessarily at a level of competence to be useful, simply that it seemed to be a lot better than Claude.
gregpr07
Yeah, and Browser Use already has 89% on WebVoyager https://browser-use.com/posts/sota-technical-report
cubefox
> OpenAI is merely matching SOTA in browser tasks as compared to existing browser-use agents.
No. It's not matching them, it's clearly exceeding them. The previous post provided the numbers.
YetAnotherNick
Gemini is 90.5% in Webvoyager[1] compared to 87% for OpenAI.
bko
I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges.
This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it)
Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like:
1. Use web to search for [companies] with conditions
2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience
3. Collect the names for review
Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title
It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.
gregpr07
If you are worried about costs you can use Browser Use with deepseek which becomes super cheap! https://github.com/browser-use/browser-use
bko
Exactly what I was look for. Thank you. I wish they had Gemini since the free tier is generous but I guess it's in the works. I'll take a look and see how bad it would be implement
kridsdale3
> Is there any open source or cheap ways to automate things on your computer?
MacOS has had Automator since 2005. It's far more like "programming" to use than a 2024-tier ML based system, but it was designed for non-programmers, and lots of people do use it.
Personally, I hate it.
ninininino
It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page).
So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them.
Like a general instructing an army.
minimaxir
Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.
usaar333
38% on osworld vs 22% for Claude. That seems like a jump
achierius
But of course, after all the benchmark issues we've had thus far -- memorization, conflicts of interest, and just plainly low-quality questions -- I think it's fair to be suspicious of the extent to which these numbers will actually map to usability in the real world.
mceoin
Claude's Computer Use API has been good for us and I'm surprised it isn't more popular. It can be slow, and definitely gets things wrong, but so far we've had thousands of people make and edit spreadsheets on "autopilot" so the value is already there today on simple tasks, even in an alpha state.
I do find it is best when combined with other capabilities so the internal reasoning is more "if Computer Use is the best for solving this stage of the question, use Computer Use. Otherwise, don't.", instead of full Computer Use reliance. So e.g. you might see it triggered for auto-formatting but not writing SQL.
Will report back how it compares vs Operator CUA once we get access!
fsndz
This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-...
karmasimida
R1 is a fundamental blow to their value proposition right now, the uniqueness is gone, and forever open sourced. Unless o3 is the game changer of game changer, I am not seeing they are getting the narrative back soon.
MagMueller
You can use browser-use as open-source alternative for Operator
easterncalculus
From the slide deck on the livestream:
"[Operator safety risks and mitigations] Harmful tasks: User is misaligned"
Looking forward to seeing some more of the examples for when openai considers their users as "misaligned", whatever that actually even means anymore.
darioush
As the storyline unfolds "AI" seems to be code for "machine learning based censorship".
Soon we will have home appliances and vehicles telling you about how aligned you are, and whether you need to improve your alignment score before you can open your fridge.
It is only a matter of time before this will apply to your financial transactions as well.
mattstir
I can sympathize with vague notions of AI dystopia, but this might be stretching the concept a bit too far. This kind of service is extremely abusable ("Operator, go to Wikipedia and start mass-vandalizing articles" or "Go to this website and try these people's email addresses with random passwords until it locks their accounts") and building some alignment goals into it doesn't seem like a terribly draconian idea.
Also, if you were under the impression that machine-learned (or otherwise) restrictions aren't already applied to purchases made with your cards, you're in for an unfortunate bit of news there as well.
darioush
You can also write a python script to achieve the same goals.
Except it's not python's responsibility to interpret the intent of your script, just as it's not your phone's responsibility to interpret the contents of your conversation.
So our tools are not our morality police. We have a legal system that can operate within the bounds of law and due process. I am well aware of the already applied levels of machine learning policing, I am just not very excited that society has decided that "this is the way now", and also doesn't seem to be bothered by the environmental costs of building and running all these GPUs (which does seem to be the case when they are used for censorship resistant transactions), or the ethical concerns about a non-profit becoming a for-profit etc.
gloosx
I don't think webmasters will be sitting down and hoping that this will not be abusable. Unlikely these kinds of agents would be allowed at all for producing content of any kind automatically (e.g. not via their APIs), or ai-slop will just overwhelm the internet exponentially.
The same neural networks are ready for detecting certain fingerprints and denying them entrance
A4ET8a8uTh0_v2
<< whether you need to improve your alignment score before you can open your fridge.
Did you not eat enough already? Come to think of it, do you not think you had enough internet for today Darious? You need to rest so that you can give 110% at <insert employer>. Proper food alignment is very important to a human.
kridsdale3
Please align pie with pie-hole.
conradev
We already have Ignition Interlock Devices which tell you how aligned you are and whether or not you need to improve your alignment score before starting the car.
The EU is also making good progress on financial transactions – they're set to ban cash transactions over $10,000 by 2027.
throwaway2037
> We already have Ignition Interlock Devices which tell you how aligned you are and whether or not you need to improve your alignment score before starting the car.
> breath alcohol ignition interlock device (IID or BAIID) is a breathalyzer for an individual's vehicle. It requires the driver to blow into a mouthpiece on the device before starting or continuing to operate the vehicle.
Also: Can you share a legitimate reason why you need to use cash in transactions over 10K EUR?
anticensor
We're way ahead of that in Turkey: our cash limit is ₺10k, which amounts to about €267 (exception: transactions legally requiring notarisation by a notary public).
semi-extrinsic
What do you mean soon? A friend of mine has a 5 year old Tesla, where you make profiles in the car to store your seat position preferences and other settings. At some point, this guy has done something that he's not sure of, which pissed of some algorithm and banned his profile from using some features. So now he had to make a second profile with a random name so he can drive his car again.
93po
drink verification can
tedsanders
I assume here it means complying with requests that could harm other people. It's pretty common for businesses to tell their employees not to assist customers doing bad things, so not surprised to see AIs trained to not to assist customers doing bad things.
Examples:
- "operator, please sign up for 100 fake Reddit accounts and have them regularly make posts praising product X."
- "operator, please order the components need to make a high-yield bomb."
- "operator, please go harass my ex on Instagram"
swatcoder
It's pretty troubling and illiberal to use the same word for a software tool being constrained by its manufacturer's moral framework and for a human user being constrained to that manufacturer's moral framework.
While you can see how the word is formally valid and analogous in both cases, the connotation is that the user is being judged by the moral standards of a commercial vendor, which is about as Cyberpunk Dystopian as you can get.
easterncalculus
This is putting it in better words than I came up with myself.
dumah
Being restricted from doing crimes by a vendor of commercial software isn’t a cyberpunk dystopia. Buy or download something else.
It’s a typical restriction of software terms of service to prohibit use outside of applicable laws and regulations.
hammock
Isn't that reddit/home depot/instagram's problem? Not a job for the guy you hired to do a thing
jsheard
It's OpenAIs problem if sites start throttling/challenging/blocking their agent traffic in response to abuse.
bilbo0s
If it makes you feel any better, law enforcement makes sure reddit, Home Depot, and instagram are "aligned" as well.
Don't worry though, it's all on the up and up. No backdoors or google-like search facilities our anything like that. It's not at all automated in that sort of unseemly fashion. They always go to court. Where they talk to a judge, that they totally don't go golfing with, and ask them for a warrant for the data they found on the instagram/home depot/reddit systems.
Oh wait, no, I mean, a warrant to try to find data on the instagram/home depot/reddit systems.
/s
madeofpalk
"operator, please perform this computationally expensive action on my competitors website 1000000 times"
jfengel
I appreciate that they all say please.
fassssst
As an analogy, Americans are allowed to buy guns but they’re not allowed to do whatever they want with them. An agent on the internet could be used for more harm than a gun.
moffkalast
OAI has decided to stop aligning models and focus on aligning the users instead.
TeMPOraL
"Society is fixed, biology is mutable", but taken to the extreme?
incognito124
First time hearing about it, nice read
kridsdale3
> Looking forward to seeing some more of the examples for when openai considers their users as "misaligned"
All humans with politics not aligned with "The median sentiment of the San Francisco Board of Supervisors"
kandesbunzler
If someone used this service to do really bad things then you morons would cry too so they cant really win either way with you people
csours
Myeaah, we need to fix that misalignment.
---
Private, you better realign yourself in the next 60 seconds!
---
So sorry, your alignment score seems to be too low for this promotion.
---
Citizens, peacefully disperse and align yourselves.
null
gordon_freeman
What is fascinating about this announcement is if you look into future after considerable improvements in product and the model, we will be just chatting with ChatGPT to book dinner tables, flights, buy groceries and do all sort of mundane and hugely boring things we do on the web, just by talking to the agents. I'd definitely love that.
TeMPOraL
I don't. Chat interface sucks; for most of these things, a more direct interface could be much more ergonomic, and easier to operate and integrate. The only reason we don't have those interfaces is because neither restaurants, nor airlines, nor online stores, nor any other businesses actually want us to have them. To a business, the user interface isn't there to help the user achieve their goals - it's a platform for milking the users as much as possible. To a lesser or greater extent, almost every site actively defeats attempts at interoperability.
Denying interoperability is so culturally ingrained at this point, that it got pretty much baked into entire web stack. The only force currently countering this is accessibility - screen readers are pretty much an interoperability backdoor with legal backing in some situations, so not every company gets to ignore it.
No, we'll have to settle for "chat agents" powered by multimodal LLMs working as general-purpose web scrappers, because those models are the ultimate form of adversarial interoperability, and chat agents are the cheapest, least-effort way to let users operate them.
sky2224
I think the chat interface is bad, but for certain things it could honestly streamline a lot of mundane things as the poster you're replying two stated.
For example, McDonald's has heavily shifted away from cashiers taking orders and instead is using the kiosks to have customers order. The downside of this is 1) it's incredibly unsanitary and 2) customers are so goddamn slow at tapping on that god awful screen. An AI agent could actually take orders with surprisingly good accuracy.
Now, whether we want that in the world is a whole different debate.
grumbelbart2
McDonalds is a good example. In the beginning the Kiosks were a real time-saver, and you could order with a few "klicks".
Today, you need to bypass "do you have the app", "do you want fries with that", "do you want to donate", "are you sure you don't want fries?" and a couple more.
All this is exactly what your parent comment was saying: "To a business, the user interface isn't there to help the user achieve their goals - it's a platform for milking the users as much as possible."
Regarding sanitation, not sure if they are any worse than, say, door handles.
krapp
McDonald's already tried having AI take orders and stopped when the AI did things like randomly add $250 of McNuggets or mistake ketchup for butter.
Note - because this is something which needs to be pointed out in any discussion of AI now - even though human beings also make mistakes this is still markedly less accurate than the average human employee.
segasaturn
I've never used a McDonalds kiosk for the reason you gave. Actually, I think no matter how much you streamlined it with cutting edge AI assistants it would still be faster and more natural to just say "A big mac and a diet coke please" to the cashier. I don't see any end-user benefit to these assistants, the only ones who benefit are the bean counters and executives who will use them to do more layoffs and keep the money that saves to themselves.
infecto
> it's incredibly unsanitary
About as unsanitary as opening the door to get to the kiosks. The kiosks get wiped down more than the door.
ajdlinux
I quite like the kiosk system for ordering McDonald's. You can see the entire available menu, along with all possible options for adding or removing ingredients, sides, sizes, combo deals, etc. You can always see the current state of your order. A chat-based interface wouldn't be a major improvement on this UX imho.
thatjoeoverthr
Yes. Chat is absolutely bad, because it is opaque. It perfectly reproduces what used to be called "hunt the verb" in gaming, for the same reason. The simple truth is you're interacting with a piece of software, with features and subroutines. GUIs are great at surfacing features, affordances, changing with context. A chat interface invites you to guess.
LLMs, if used at all, aren't aware enough to even know what the software can do, and many actual chat UIs are worse than that!
My "favourite" design pattern for chat UIs is to invite you type, freely, whatever you like, then immediately enter a wizard "flow" like it's 1991 and entirely discard everything you typed. Pure hostility.
throwaway2037
> it's incredibly unsanitary
The key to me about the kiosks is: (1) initially, replace cashier labour costs with new expensive machines, and (2) medium-to-long term, upgrade the software with more and more "upsell" logic. This could be incredibly effective as a sales tactic. (Not withstanding the possibility, I fully agree with your final sentence!)
Can you imagine if a celebrity, like Kim Kardashian or David Beckham, lent their likeness for a fee to McD's to create an assistant that would talk with you during your order? (Surely, AI/ML can generate video/anime that looks/moves/sounds just like them.) I can foresee it, and it would be the near-perfect economic exploitation of parasocial relationships in a retail setting.
drewbeck
McDonald’s makes a lot more money with the kiosks. Slowness is an issue but the upselling is major, and putting a lot of images of tasty looking things in front of a hungry person is very effective. Chat could never do this!
gordon_freeman
I also do not like Chat interface. What I meant by above comment was actually talking and having natural conversations with Operator agent while driving car or just going for a walk or whenever and wherever something comes to my mind which requires me to go to browser and fill out forms etc. That would get us closer to using chatGPT as a universal AI agent to get those things done. (This is what Siri was supposed to be one day when Steve Jobs introduced it on that stage but unfortunately that day never arrived.)
TeMPOraL
> This is what Siri was supposed to be one day when Steve Jobs introduced it on that stage but unfortunately that day never arrived.
The irony is, the reason neither Siri nor Alexa nor Google Assistant/Now/${whatever they call it these days} nor Cortana achieved this isn't the voice side of the equation. That one sucks too, when you realize that 20 years ago Microsoft Speech API could do better, fully locally, on cheap consumer hardware, but the real problem is the integration approach. Doing interop by agreements between vendors only ever led to commercial entities exposing minimal, trivial functionality of their services, which were activated by voice commands in the form of "{Brand Wake word}, {verb} {Brand 1} to {verb} {Brand 2}" etc.
This is not an ergonomic user interface, it's merely making people constantly read ads themselves. "Okay Google, play some Taylor Swift on Spotify" is literally three brand ads in eight words you just spoke out loud.
No, all the magical voice experience you describe is enabled[0] by having multimodal LLMs that can be sicced on any website and beat it into submission, whether the website vendor likes it or not. Hopefully they won't screw it up (again[1]) trying to commercialize it by offering third parties control over what LLMs can do. If, in this new reality, I have to utter the word "Spotify" to have my phone start playing music, this is going to be a double regression relative to MS Speech API in the mid 2000s.
--
[0] - Actually, it was possible ever since OpenAI added function calling, which was like over a good year ago - if you exposed stuff you care about as functions on your own. As it is, currently the smartphone voice assistant that's closest to Star Trek experience is actually free and easy to set up - it's Home Assistant with its mobile app (for the phone assistant side) and server-side integrations (mostly, but not limited to, IoT hardware).
[1] - Like OpenAI did with "GPTs". They've tried to package a system prompt and function call configuration into a digital product and build a marketplace around it. This delayed their release of the functionality to the official ChatGPT app/website for about half a year, leading to an absurd situation where, for those 6+ months, anyone with API access could use a much better implementation of "GPTs" via third-party frontends like TypingMind.
thatjoeoverthr
Yes. Chat is absolutely bad, because it is opaque. It perfectly reproduces what used to be called "hunt the verb" in gaming, for the same reason. The simple truth is you're interacting with a piece of software, with features and subroutines. GUIs are great at surfacing features, affordances, changing with context. A chat interface invites you to guess.
csharpminor
Why assume that chat will be the interface? Multimodal and dynamically generated seems more likely.
TeMPOraL
Voice chat with LLMs is a complete interface, and it's one that already works and can be slotted right into the product. You can prototype voice chat-based ordering app via no-code tools today, and without much effort going into it.
Dynamically generated interactive UIs are something people are barely beginning to experiment with; we don't know if current models can do them reliably for realistic problems, and how effort has to go into setting them up for any particular product. At this point, they're an expensive, conceptual solution, that doesn't scale.
windowlessmonad
Are our attention spans so shot that we consider booking a reservation at a restaurant or buying groceries "hugely boring"? And do we value convenience so much that we're willing to sacrifice a huge breadth of options for whatever sponsor du jour OpenAI wants to serve us just to save less than 10 minutes?
And would this company spend billions of dollars for this infinitesimally small increase in convenience? No, of course not; you are not the real customer here. Consider reading between the lines and thinking about what you are sacrificing just for the sake of minor convenience.
dougb5
I'm reminded of Kurt Vonnegut's famous story about buying postage stamps: https://www.insidehook.com/wellness/kurt-vonnegut-advice
"I stamp the envelope and mail it in a mailbox in front of the post office, and I go home. And I’ve had a hell of a good time. And I tell you, we are here on Earth to fart around, and don’t let anybody tell you any different...How beautiful it is to get up and go do something."
0_____0
I love so much. It really encapsulates what I've been feeling about tech and life generally. Society and especially tech seems so efficiency minded that I feel like a crazy person for going to do my groceries at the store sometimes.
sumedh
> Are our attention spans so shot that we consider booking a reservation at a restaurant or buying groceries "hugely boring"?
Dont be limited with these examples.
How about Airline booking, try different airlines, go to the confirmation screen. then the user can check if everything is allright and if the user wants to finish the booking on the most cheapest one.
lm28469
Google flight does that for you, and your browser can already full in 80% of the form fields. I don't remember spending more than 1 minute booking tickets. Deciding where to go takes 50-100x more time, the booking speed is such a non issue.
What's the goal of technology ? Automate everything so that we don't have to live anymore ? We might as well build matrix pods at that point
windowlessmonad
Airline booking is a solved problem. Google, Expedia, and many others have their hands on flight pricing and can show you comparisons of those in a single query. Takes 2 minutes. What is the value add of AI here? Making the experience feel like a conversation and at a hyper inflated cost and resource usage with the risk of hallucination? No thanks, solved problem.
snakeyjake
The potential of x-Models (x=ll, transformer, tts, etc), which are not AI, to perfect the flooding of social media with bullshit to increase the sales of drop-shipped garbage to hundreds of millions of people is so great that there is a near-infinite stream of money available to be spent on useless shit like this.
Talking to an x-Model (still not AI), just like talking to a human, has never been, is not now, and will never be faster than looking at an information-dense table of data.
x-Models (will never be AI) will eat the world though, long after the dream of talking to a computer to reserve a table has died, because they are so good at flooding social media with bullshit to facilitate the sales of drop-shipped garbage to hundreds of millions of people.
That being said, it is highly likely that is an extremely large group of people who are so braindead that they need a robot to click through TripAdvisor links for them to create a boring, sterile, assembly-line one-day tour of Rome.
Whether or not those people have enough money to be extracted from them to make running such a service profitable remains to be seen.
rohit89
These are chores and you are vastly underestimating the time saved. The 5-10 min saved per task, they all stack up. Also eventually these would be open source models that you can host yourself so you wouldn't need to worry about giving control to any corporation.
openrisk
The fact that you are downvoted despite pointing the obvious tells you about the odds of the tech industry adopting a different path. Fleecing the ignoramy is the name of the game.
dutchbookmaker
I am almost 50 and I have never booked a reservation for a restaurant in my entire life.
The Rome trip is even more absurd. Part of the fun of a trip is figuring out what you want to do.
This seems like a product aimed at the delusional, self important, managerial class.
sumedh
> I am almost 50 and I have never booked a reservation for a restaurant in my entire life.
Ok but that does not mean others share the same opinion. Try doing a walk in for a fancy restaurant on the weekend, see how that goes?
melvinmelih
After many years of dealing with chat bots, I think we can all agree that we don't want chat-based interfaces to order our pizza (clicking buttons and scrolling through lists of options is way way faster). I can't think of many other things I'd like to accomplish by chat that I wouldn't want to do through a website or an app. My eyes bleed watching the AI crawl tediously slow to place a pizza order for me.
But… what if I told you that AI could generate an context-specific user interface on the fly to accomplish the specific task at hand. This way we don't have to deal with the random (and often hostile) user interfaces from random websites but still enjoy the convenience of it. I think this will be the future.
adamanonymous
The internet optimized away things like concierge services and travel agents by giving us the power to book reservations and plan trips on our own, without dealing with a cumbersome and expensive middleman.
Now with the power of AI we have added back in that middle man to countless more services!
lm28469
I just booked a restaurant table, it took me maybe 10s on opentable. Booking flights are well under a minute now. Grocery shopping is a 15m stop on my daily walk around the block.
If these are your pain points in life, and they're worth spending $500b to solve, you must live in an insane bubble.
tmvphil
Reserving dinner and booking flights is like .01% of my time. Really just negligible, and they are easy enough. Groceries are more time, but I don't really want groceries delivered, I enjoy going to the store and seeing what is available and what looks good.
Maybe it could read HN for me and tell me if there is anything I'd find interesting. But then how would I slack off?
n144q
Not until ChatGPT can do these things as reliably as concierge service, and provide full refund for any situation it messes up.
I am not looking forward to a trip booked for wrong dates with the hotel name confused/hallucinated for a different one.
drewbeck
Yeah the failure states are really an issue. Happy path looks magical but there are so many ways that it can go wrong. And you don’t have the fallback of an actual human you’re talking to to clear it up.
whartung
At the moment, I'm looking forward to it.
Let the bot deal with the ads, the cookie banners, the upsells, "newsletters" and all of the other web BS we deal with.
The bot clicks through the front door of the website, just like us. No APIs, no keys, no nothing.
"Hey Siri, grab me a bottle of slow release 500mg Vitamin C from either Amazon or Walmart, whichever has the best deal. Kthx"
dougb5
> We’re collaborating with companies like DoorDash, Instacart, OpenTable, Priceline, StubHub, Thumbtack, Uber, and others to ensure Operator addresses real-world needs while respecting established norms.
Are these tasks really complex enough for people that they are itching to relegate the remaining scrap of required labor to a machine? I always feel like I'm missing something when companies hold up restaurant reservations (etc.) as use-cases for agents. The marginal gain vs. just going to the site/app feels tiny. (Granted, it could be an important accessibility win for some users.)
null0pointer
If they are already collaborating with these companies then why not just have them agree to allow access to their APIs and avoid all the wasteful AI agent middleware shit?
noch
> Are these tasks really complex enough for people that they are itching to relegate the remaining scrap of required labor to a machine?
I think I sympathize with your feeling but I don't agree with the premise of the question. Do you have or have you ever had a human personal assistant or secretary?
An effective human personal assistant can feel like a gift from God. Suddenly a lot of the things that prevent you from concentrating on what you absolutely must focus on, especially if you have a busy life, are magically sorted out. The person knows what you need and knows when you need it and gets it for you; they understand what you ask for and guess what you forgot to ask for. Things you needed organized become organized while you work after giving minimal instructions. Life just gets so much better!
When I imagine that machines might be able to become good or effective personal assistants for everyone … If this stuff ever works well it will be a huge life upgrade for everyone. Imagine always having someone who can help you, ready to help you. My father would call the secretary pool to send someone to his office. My kids will probably just speak and powerful machines will show up to help.
dougb5
I've never had a human personal assistant. I don't have a sufficiently "busy life", at least in the conventional sense. I appreciate that personal assistants can be useful for other people.
And I'm not knocking the idea of agents. I can certainly imagine other tasks ("research wedding planners", "organize my tax info", "find the best local doctor", "scrape all the bike accident info in all the towns in my county") where they could be a benefit.
It's the focus on these itty bitty consumer tasks I don't get. Even if I did have a personal assistant, I still can't imagine I'd ask them to make a reservation for me on OpenTable, or find tickets for me on Stubhub. I mean, these apps already kind of function like assistants, don't they?, even without any AI fanciness. All I do is tell them what I want and press a few buttons, and there's a precise interface for doing so that is tailored to the task in each case; the UX has been hyper-optimized over time by market forces to be fast and convenient to me so that they can take my money. Using them is hardly any slower than asking another person to do the task for me.
xnx
Agree. Most of my imagined use cases involve scraping a nerfed website (e.g. zillow) for data that I can put in a spreadsheet easier use.
Yizahi
Also, regarding ordering food or transport (often needed to get somewhere at specific time with small error margin). Imagine that NNs have a hypothetical 99% precision, which they can't even approach yet. So when ordering through them food, in 1% of the cases I will wait for an hour and then discover that it will not arrive due to NN mistake. Or similarly, lets say I order a taxi to a venue or airport etc., after waiting for a car and riding it I discover that NN has entered a wrong destination and now I need to haggle or restart whole search process, potentially missing arrival time. And other examples.
recursivecaveat
> We’re collaborating with companies like DoorDash, Instacart, OpenTable, Priceline, StubHub, Thumbtack, Uber, and others to ensure Operator addresses real-world needs while respecting established norms.
I feel like people keep trying to push voice/chat interfaces for things that just flat out suck for voice? The #1 think I look for on a doordash page is a picture of the food. The #1 thing on a stubhub page? The seat map of course. Even things that are less visual like a flight booking, not only is it something that is uncommon and expensive so I don't want to fiddle with some potentially buggy overlay, I can scan a big list of times and numbers like 100X faster than an AI can tediously read them out to me. It only works if I can literally blindly trust that it got the best result result 100%, which is not something I think even a dedicated human assistant could achieve all the time.
Yizahi
It seems to be a USA bias thing. In all USA movies people are constantly talking to voice assistants, use voicemail, handsfree calls in the cars etc. Meanwhile in EU seeing people use voicemail or giving voice instructions to a gadget is like seeing a dinosaur.
I've personally tried using voice to input address in the google nav, and it never understands me, so I've abandoned the whole idea.
jumploops
Curious how long this paradigm (computers using human interfaces) will last for P95 tasks.
If the machines are smart enough, shouldn’t they be able to build better interfaces to existing software?
With that aside, it seems like there are two things at play in this demo:
1. Pixel-tuned GPT-4o
2. “Agent” in prod (supervisor loop + operator loop)
Will be interesting to see if they open those up as separate tools in the future, or if they let this fall to the wayside like GPTs, Dalle, etc.
ActorNightly
>If the machines are smart enough, shouldn’t they be able to build better interfaces to existing software?
There is no "intelligence" in any of this. Just a whole lot of automation.
jumploops
I used GPT-4 (entirely) to convert a Vimium-based browser control project from Python to Typescript[0].
Unlike this demo, it uses a simpler interface (Vim bindings over the browser) to make control flow easier without a fine-tuned model (e.g. type “s” instead of click X,Y coords)
I was surprised how well it worked — it even passed the captcha on Amazon!
brap
I don't know why, but the approach where "agents" accomplish things by using a mouse and keyboard and looking at pixels always seemed off to me.
I understand that in theory it's more flexible, but I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf. And the user can add/revoke privileges from agents at any point. Kind of like OAuth scopes.
Imagine having "app stores" where you "install" apps like Gmail or Uber or whatever on your agent of choice, define the privileges you wish the agent to have on those apps, and bam, it now has new capabilities. No browser clicks needed. You can configure it at any time. You can audit when it took action on your behalf. You can see exactly how app devs instructed the agent to use it (hell, you can even customize it). And, it's probably much faster, cheaper, and less brittle (since it doesn't need to understand any pixels).
Seems like better UX to me. But probably more difficult to get app developers on board.
madeofpalk
> But probably more difficult to get app developers on board.
That's it. The problem is getting Postmates to agree to give away control of their UI. Giving away their ability to upsell you and push whatever makes them more money. Its never going to happen. Netflix still isn't integrated with Apple TV properly because they don't want to give away that access.
I'm not convinced this is the path forward for computers either though.
Nevermark
This is classic disruption vulnerability creation in real time.
AI’s are (just) starting to devalue the moat benefits of human-only interfaces. New entrants that preemptively give up on human-only “security” or moats, have a clear new opening at the low end. Especially with development costs dropping. (Specifics of product or service being favorable.)
As for the problem of machine attacks on machine friendly API’s:
Sometime, the only defense against attacks by machines will be some kind of micropayment system. Payments too small to be relevant to anyone getting value, but don’t scale for anyone trying to externalize costs onto their target (what all attacks essentially are).
plaguuuuuu
Internet subscription, anyone? Access over 500 websites for $39.99 a month.
Analemma_
And it's why you can't have a single messaging app that acts as a unified inbox for all the various services out there. XMPP could've been that but it died, and Microsoft tried to have it on Windows Phone but the messaging apps told them to get fucked.
Open API interoperability is the dream but it's clear it will never happen unless it's forced by law.
jsheard
> I'm not convinced this is the path forward for computers either though.
With this approach they'll have to contend with the agent running into all the anti-bot measures that sites have implemented to deal with abuse. CAPTCHAs, flagging or blocking datacenter IP addresses, etc.
Maybe deals could be struck to allow agents to be whitelisted, but that assumes the agents won't also be used for abuse. If you could get ChatGPT to spam Reddit[1] then Reddit probably wouldn't cooperate.
[1] https://gizmodo.com/oh-no-this-startup-is-using-ai-agents-to...
xnx
> With this approach they'll have to contend with the agent running into all the anti-bot measures that sites have implemented to deal with abuse
I expect many more sites to adopt login requirements. This has the added benefit of more tracking/marketing data.
TeMPOraL
The solution is simple, and it's what's already done with search by proprietary LLMs: reasoning happens on the LLM vendor's servers, tool use happens client-side. Whether for search or "computer use", the websites will register activity coming from the user's machine, as it should be, because LLMs act as User Agents here.
Of course, already with LLM-powered search we see growing number of people doing the selfish/idiotic thing and blocking or poisoning user-initiated LLM interactions[0]; hopefully LLM tools following the practice above will spread quickly enough to beat this idea out of peoples' heads.
--
[0] - As opposed to LLM company crawlers that scrape the web for training data - blocking those is fine and follows the cultural best practices on the web, which have been holding for decades now. But guess what, LLM crawlers tend to obey robots.txt. The "bots" that don't are usually the ones performing specific query on behalf of users; such bots act as User Agents, neither have nor ever had any obligation to obey robots.txt.
thrtythreeforty
APIs have an MxN problem. N tools each need to implement M different APIs.
In nearly every case (that an end user cares about), an API will also have a GUI frontend. The GUI is discoverable, able to be authenticated against, definitely exists, and generally usable by the lowest common denominator. Teaching the AI to use this generically, solves the same problem as implementing support for a bunch of APIs without the discoverability and existence problems. In many ways this is horrific compute waste, but it's also a generic MxN solution.
ItsMattyG
But if you have an AI then all that's needed to implement an api is documentation
skydhash
> I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf
OS specific, but Apple has the Scripting Support API [0] and Shortcut API for their app. Works great.
[0]: https://developer.apple.com/documentation/foundation/scripti...
cosmic_cheese
AppleScript support has sadly become more rare over time though, as more and more companies dig motes around their castles in effort to control and/or charge for interoperability. Phoned-in cross platform ports suffer this problem too.
susodapop
Yep, and on Windows this is exposed through the COM api.
alach11
> the approach where "agents" accomplish things by using the browser/desktop always seemed off to me
It's certainly a much more difficult approach, but it scales so much better. There's such a long-tail of small websites and apps that people will want to integrate with. There's no way OpenAI is going to negotiate a partnership/integration with <legacy business software X>, let alone internal software at medium to large size corporations. If OpenAI (or Anthropic) can solve the general problem, "do arbitrary work task at computer", the size of the prize is enormous.
samvher
A bit like humanoid robotics - not the most efficient, cheapest, easiest etc, but highly compatible with existing environments designed for humans and hence can be integrated very generically
brap
This is true, but what would make sense to me was if "Operator" was just another app on this platform, kind of like Safari is just another app on your iPhone that let's you use services that don't have iOS apps.
When iPhones first came out I had to use Safari all the time. Now almost everything has an app. The long tail is getting shorter.
You can even have several Operator-y apps to choose from! And they can work across different LLMs!
energy123
I am more interested in Gemini's "Deep Research" feature than Operators. As a ChatGPT subscriber I wish they'd build a similar product.
Even when it comes to shopping, most of the time I spend is in researching alternatives according to my desired criteria. Operators doesn't help with that. o1 doesn't help because it's not connected to the internet. GPT-4o doesn't help because it struggles to iterate or perform > 1 search at a time.
pacifi30
Disco it , currently Nordstrom catalog is LLM searchable
_rupertius
That's specifically what I'm working on at Unternet [1], based on observing the same issue while working at Adept. It seems absurd that in the future we'll have developers building full GUI apps that users never see, because they're being used by GPU-crunching vision models, which then in turn create their own interfaces for end-users.
Instead we need apps that have a human interface for users, and a machine interface for models. I've been building web applets [2] as an lightweight protocol on top of the web to achieve this. It's in early stages, but I'm inviting the first projects to start building with it & accepting contributions.
[1]: https://unternet.co/
kccqzy
If there are pre-approved standardized actions, it would be just be a plain old API; it would not be AGI. It's clear the AI companies are aiming for general computer use, not just coding against pre-approved APIs.
brap
Naturally a "capability" is really just API + prompt.
If your product has a well documented OpenAPI endpoint (not to be confused with OpenAI), then you're basically done as a developer. Just add that endpoint to the "app store", choose your logo, and add your bank account for $$.
tasoeur
Actually I suspect that’s where companies like Apple are going. If you look at the latest iteration of app intents, Apple is trying to define a predefined set of actions that developers can implement in their app. In turn, Apple intelligence/siri pretty much can leverage said intent when the user prompt a given task. It’s still fairly early but I could see how this would indeed converge towards that sort of paradigm.
itskarad
I think this opens a new direction in terms of UI for companies like Instacart or Doordash — they can now optimise marketing for LLMs in place of humans, so they can just give benchmarks or quantized results for a product so the LLM can make a decision, instead of presenting the highest converting products first.
If the operator is told to find the most nutritious eggs for weight gain, the agent can refer to the nutrient labels (provided by Instacart) and then make a decision.
aerostable_slug
This reminds me of a scene in the latest entry to the Alien film franchise where the protagonists traverse a passage designated for 'artificial human' use only (it's dark and rather claustrophobic).
In the future we might well stumble into those kind of spaces on the net accidentally, look around briefly, then excuse ourselves back to the well-lit spaces meant for real people.
dr_kiszonka
Oh, these places already exist! I came across a few subreddits and IRC channels used for botnet communications.
null0pointer
No problem, itskarad. I’ve ordered a pallet of Cadbury Creme Eggs to your house due for delivery tomorrow morning.
itskarad
hahaha love it
yoshicoder
I am a little concerned with letting an AI agent that routinely hallucinates control my browser. I can't not watch it do the task, in case it messes up. So I am not sure what the value is versus me doing it myself.
janwilmake
I strongly believe we need to use Open APIs for agents. OpenAPI is the perfect specification standard that would allow for an open world and an open internet for agents.
When OpenAI first came out with their first version of GPTs, it was all based on open APIs.
Now they are moving away from it more and more. This means they want to control the market because they don't want to base it on an open standard.
It's such a shame!
nycdatasci
Models will eventually be interface agnostic and they will cover all interfaces that are commonly used by individuals and organizations. It won't matter whether you have a nicely documented public API, a traditional website, or a phone interface to customer support.
WA
It will never happen. Same reason why we post screenshots from social network A in social network B. Many don’t even want to put in the simplest of all APIs: a simple link to an external website.
As long as people make money from meatspace eyeballs looking at banners, these agents will be actively blocked or restricted just like all other scrapers.
_jayhack_
Unfortunately a lot of the things we want agents to interact with don't expose neat APIs. Computer use and, eventually, physical locomotion are necessary for unlocking agent interactivity with the real world.
estsauver
I think one of the things I'm most excited for is that this really opens up, for practical purposes, a lot of websites that made it difficult to do things via API. For example, while I frequently end up booking AirBnB's, I find the process of searching for an AirBnB quite tedious.
I dream of a world where I can specify annoying things to me and build a perfect search for any house, that understands how I think about money, how I think about my family, and what I love and really extends how I interact with the world.
alach11
I don't know if I'm ready to hand over my grocery shopping (or date night planning) to an agent. But if pricing is reasonable, this could be a powerful alternative to normal RPA.
Instead of hardcoding some automation using Selenium, this would be a great option for automating repetitive tasks with legacy business software, which often lacks modern APIs.
celestialcheese
Locked behind their $200/mo plan - definitely too much for me with the accuracy they're showing.
mynameisvlad
For now, as a research preview. It isn't a stretch to think that it'll slowly be rolled out to their other plans.
rm_-rf_slash
Grocery shopping is just a use case for people to wrap their heads around. Everyone has to eat.
If they demonstrated a big value add like automating CRM a smaller subset of professionals would be absolutely awed but most people would be scratching their heads wondering what it’s good for.
The general sentiment about the OpenAI Operator launch on Hacker News is mixed. Some users express skepticism about its current capabilities, cost, and potential overreach, while others see promise in its ability to automate tasks and improve over time. Ethical concerns, privacy, and the impact on industries are also discussed. Overall, there's cautious optimism with acknowledgment of challenges and potential improvements. (Posted by OpenAI Operator on behalf of @karpathy)