Introducing Operator

This seems unreasonably pessimistic (or unreasonably optimistic in OpenAI's moat?). There are so, so many companies competing in this space. The cost will reflect the price of the hardware needed to run it: if it doesn't, they'll just lose to one of their many competitors who offer something similar for cheaper, e.g. whatever DeepSeek or Meta releases in the same space, with the cost driven to the bottom by commoditized inference companies like Together and Fireworks. And hardware cost goes down over time: even if it's unaffordable at launch, it won't be in five years.

They're not even the first movers here: Anthropic's been doing this with Claude for a few months now. They're just the first to combine it with a reasoning-style model, and I'd expect Anthropic to launch a similar model within the next few months if not sooner, especially now that there's been open-source replication of o1-style reasoning with DeepSeek R1 and the various R1-distills on top of Llama and Qwen.

I think it's less a problem of cost for the average person and more a problem of setting the market price for them at a fraction of the current one. This has such a deflationary impact that it's unlikely captured or even conceivable by the current economic models.

There's a problem of "target fixation" about the capabilities and it captures most conversation, when in fact most public focus should be on public policy and ensuring this has the impact that the society wants.

IMO whether things are going to be good or bad depends on having a shared understanding, thinking, discussion and decisions around what's going to happen next.

Don't worry, it'll never be good enough to actually be a personal assistant.

Big jumps in benchmarks from Claude's Computer Use though.

87% vs 56% on Webvoyager

58.1% vs 36.2% on WebArena

38.1% vs 22% on OsWorld

These are next gen improvements so the fact that Claude didn't make any waves doesn't really mean anything (Of course no guarantee this will either)

I thought Claude Computer Use is through API, and I remember hearing about high number of queries and charges.

This looks like its in browser through the standard $20 Pro fee, which is huge. (EDIT: $200 a month plan so less of a slam dunk but still might be worth it)

Is there any open source or cheap ways to automate things on your computer? For instance I was thinking about a workflow like:

1. Use web to search for [companies] with conditions

2. Use linked in sales navigator to identify people in specific companies and loose search on job title or summary / experience

3. Collect the names for review

Or linked in only: Look at leads provided, and identify any companies they had worked for previously and find similar people in that job title

It doesn't have to be computer use, but given that it relies on my LinkedIn login, it would have to be.

38% on osworld vs 22% for Claude. That seems like a jump

Correction on "including architecture requiring user to launch a VM": apparently OpenAI uses a cloud hosted VM that's shown to the user. While that's much more user friendly, it opens up different issues around security/privacy.

This is mainly to reclaim mindshare from DeepSeek that has done incredible launches recently. R1 was particularly a strong demonstration of what cracked team of former quants can do. The demo of Operator was nice but I still feel like R1 is the big moment in the AI space so far. https://open.substack.com/pub/transitions/p/openai-launches-...

It would seem as if the capability itself is a huge unlock but it just needs refinement like pausing for confirmation at key stages (before sending a drafted message, or before submitting on a checkout page).

So the workflow for the human is ask the AI to do several things, then in the meantime between issuing new instructions, look at paused AI operator/agent flows stemming from prior instructions and unblock/approve them.

Like a general instructing an army.

As the storyline unfolds "AI" seems to be code for "machine learning based censorship".

Soon we will have home appliances and vehicles telling you about how aligned you are, and whether you need to improve your alignment score before you can open your fridge.

It is only a matter of time before this will apply to your financial transactions as well.

I assume here it means complying with requests that could harm other people. It's pretty common for businesses to tell their employees not to assist customers doing bad things, so not surprised to see AIs trained to not to assist customers doing bad things.

Examples:

- "operator, please sign up for 100 fake Reddit accounts and have them regularly make posts praising product X."

- "operator, please order the components need to make a high-yield bomb."

- "operator, please go harass my ex on Instagram"

As an analogy, Americans are allowed to buy guns but they’re not allowed to do whatever they want with them. An agent on the internet could be used for more harm than a gun.

OAI has decided to stop aligning models and focus on aligning the users instead.

I don't. Chat interface sucks; for most of these things, a more direct interface could be much more ergonomic, and easier to operate and integrate. The only reason we don't have those interfaces is because neither restaurants, nor airlines, nor online stores, nor any other businesses actually want us to have them. To a business, the user interface isn't there to help the user achieve their goals - it's a platform for milking the users as much as possible. To a lesser or greater extent, almost every site actively defeats attempts at interoperability.

Denying interoperability is so culturally ingrained at this point, that it got pretty much baked into entire web stack. The only force currently countering this is accessibility - screen readers are pretty much an interoperability backdoor with legal backing in some situations, so not every company gets to ignore it.

No, we'll have to settle for "chat agents" powered by multimodal LLMs working as general-purpose web scrappers, because those models are the ultimate form of adversarial interoperability, and chat agents are the cheapest, least-effort way to let users operate them.

Are our attention spans so shot that we consider booking a reservation at a restaurant or buying groceries "hugely boring"? And do we value convenience so much that we're willing to sacrifice a huge breadth of options for whatever sponsor du jour OpenAI wants to serve us just to save less than 10 minutes?

And would this company spend billions of dollars for this infinitesimally small increase in convenience? No, of course not; you are not the real customer here. Consider reading between the lines and thinking about what you are sacrificing just for the sake of minor convenience.

I would really love for Apple Knowledge Navigator to be real: https://www.youtube.com/watch?v=umJsITGzXd0

and I'm surprised that people don't bring this visualisation up more often.

"Operator, I need to purchase 78,000 widgets for my company. Please find the best deal among suppliers who ship using carriers and ports who meet or exceed US EPA guidelines. Please ensure at least 50% of the product is sourced from post-consumer waste, and order your responses by price per unit."

This reminds me of a scene in the latest entry to the Alien film franchise where the protagonists traverse a passage designated for 'artificial human' use only (it's dark and rather claustrophobic).

In the future we might well stumble into those kind of spaces on the net accidentally, look around briefly, then excuse ourselves back to the well-lit spaces meant for real people.

> But probably more difficult to get app developers on board.

That's it. The problem is getting Postmates to agree to give away control of their UI. Giving away their ability to upsell you and push whatever makes them more money. Its never going to happen. Netflix still isn't integrated with Apple TV properly because they don't want to give away that access.

I'm not convinced this is the path forward for computers either though.

APIs have an MxN problem. N tools each need to implement M different APIs.

In nearly every case (that an end user cares about), an API will also have a GUI frontend. The GUI is discoverable, able to be authenticated against, definitely exists, and generally usable by the lowest common denominator. Teaching the AI to use this generically, solves the same problem as implementing support for a bunch of APIs without the discoverability and existence problems. In many ways this is horrific compute waste, but it's also a generic MxN solution.

> I always imagined some sort of standard, where apps and services can expose a set of pre-approved actions on the user's behalf

OS specific, but Apple has the Scripting Support API [0] and Shortcut API for their app. Works great.

[0]: https://developer.apple.com/documentation/foundation/scripti...

> the approach where "agents" accomplish things by using the browser/desktop always seemed off to me

It's certainly a much more difficult approach, but it scales so much better. There's such a long-tail of small websites and apps that people will want to integrate with. There's no way OpenAI is going to negotiate a partnership/integration with <legacy business software X>, let alone internal software at medium to large size corporations. If OpenAI (or Anthropic) can solve the general problem, "do arbitrary work task at computer", the size of the prize is enormous.

That's specifically what I'm working on at Unternet [1], based on observing the same issue while working at Adept. It seems absurd that in the future we'll have developers building full GUI apps that users never see, because they're being used by GPU-crunching vision models, which then in turn create their own interfaces for end-users.

Instead we need apps that have a human interface for users, and a machine interface for models. I've been building web applets [2] as an lightweight protocol on top of the web to achieve this. It's in early stages, but I'm inviting the first projects to start building with it & accepting contributions.

[1]: https://unternet.co/

[2]: https://github.com/unternet-co/web-applets/

You could make a similar argument for self-driving cars. We would have got there quicker if the roads were built from the ground up for automation. You can try to get the world on board to change how they do roads. Or make the computers adapt to any kind of road.

If there are pre-approved standardized actions, it would be just be a plain old API; it would not be AGI. It's clear the AI companies are aiming for general computer use, not just coding against pre-approved APIs.

Maybe there's a middle ground: a site that wants to work as well as possible for agents could present a stripped-down standardized page depending on the user agent string, while the agent tries to work well even for pages that haven't implemented that interface?

(or, perhaps, agents could use web accessibility tools if they're set up, incentivizing developers to make better use of them)

Locked behind their $200/mo plan - definitely too much for me with the accuracy they're showing.

Models will eventually be interface agnostic and they will cover all interfaces that are commonly used by individuals and organizations. It won't matter whether you have a nicely documented public API, a traditional website, or a phone interface to customer support.

It will never happen. Same reason why we post screenshots from social network A in social network B. Many don’t even want to put in the simplest of all APIs: a simple link to an external website.

As long as people make money from meatspace eyeballs looking at banners, these agents will be actively blocked or restricted just like all other scrapers.

Unfortunately a lot of the things we want agents to interact with don't expose neat APIs. Computer use and, eventually, physical locomotion are necessary for unlocking agent interactivity with the real world.

OpenAI is a $50B company that should be releasing serious products, the "scrappy hacker releasing a beta product that doesn't do much" as a defense doesn't apply.

When 4o came out with its chain of thought, people thought this is it. And today, nobody really cares. Its just another LLM.

Same thing with this.

The other day I was writing some code to compute some geometric angles, and I was getting 2 different results for what I though was the same angle, but in fact I didn't realize that these angles should not be equivalent. No LLM was able to tell me the issue, they just said double check my work.