AI founders will learn the bitter lesson

Yup. And it’s already playing out that way. Anthropic, OpenAI, Gemini - technically not an upstart. All have hyperscalers backing and subsidizing their model training (AWS, Azure, GCP, respectively). It’s difficult to discern where the segmentation between compute and models are here.

    > AI will not play out the way the Web did (upstarts unseat giants) 

I recently spoke to a doctor that wanted to do a startup one part of which is an AI agent that can provide consumers second opinions for medical questions. For this to be safe, it will require access to not only patient data, but possibly front line information from content origins like UpToDate because that content is a necessity to provide grounded answers for information that's not in the training set and not publicly available via search.

The obvious winner is UpToDate who owns that data and the pipeline for originating more content. If you want to build the best AI agent for medical analysis, you need to work with UpToDate.

    > ...not brand new startups who have to try to convince companies to give them all their data for the first time.

> will instead play out like smartphones (giants entrench and balloon).

Someone correct me if I'm wrong, but didn't smartphones go the "upstarts unseat giants" way? Apple wasn't a phone-maker, and became huge in the phone-market after their launch. Google also wasn't a phone-maker, yet took over the market slowly but surely with their Android purchase.

I barely see any Motorola, Blackberry, Nokia or Sony Ericsson phones anymore, yet those were the giants at one time. Now it's all iOS/Android, two "upstarts" initially.

The people selling models are actually just selling compute

Yes, fully agreed. Anything AI is discovering in your dataset could have been found by humans, and it could have been done by a more efficient program. But that would require humans to carefully study it and write the program. AI lets you skip the novel analysis of the data and writing custom programs by using a generalizable program that solves those steps for you by expending far more compute.

I see it as, AI could remove the most basic obstacle preventing us from applying compute to vast swathes of problems- and that’s the need to write a unique program for the problem at hand.

> All competitors at this point know how important data is, and they're not going to sell it to a startup when they could package it up themselves.

Except they won't package it themselves because they are inept and inert. They still won't sell it to startups though.

This really puzzles me. I tried Cursor and was completely underwhelmed. The answers it gave (about a 1.5M loc messy Spring codebase) were surface-level and unhelpful to anyone but a Java novice. I get vastly better work out of my intern.

To add insult to injury, the IntelliJ plugin threw spurious errors. I ended up uninstalling it and marking my calendar to try again in 6 months.

Yet some people say Cursor is great. Is it something about my project? I can't imagine how it deals with a codebase that is many millions of tokens. Or is it something about me? I'm asking hard questions because I don't need to ask the easy ones.

What are people who think Cursor is great doing differently?

So isn’t cursor just a tool for Claude or ChatGpt to use? Another example would be a flight booking engine. So why can’t an AI just talk direct to an IDE? This is hard as the process has changed, due to the human needing to be in the middle.

So Isn’t AI useless without the tools to manipulate?

I’m very “bullish” on AI in general but find cursor incredibly underwhelming because there is little value add compared to basically any other AI coding tool that goes beyond autocomplete. Cursor emphatically does not understand large codebases and smaller (few file codebases) can just be pasted into a chat context in the worst case.

Is it really that different to Claude with tools via MCP, or my own terminal-based gptme? (https://github.com/ErikBjare/gptme)

It comes down to moats. Does OpenAI have a moat? It's leading the pack, but the competitors always seem to be catching up to it. We don't see network effects with it yet like with social networks, unless OpenAI introduces household robots for everyone or something, builds a leading marketshare in that segment, and the rich data from these household bots is enough training data that one can't replicate with a smaller robot fleet.

And AI is too fundamental of a technology that a "loss leader biggest wallet wins" strategy, used by the likes of Uber, will work.

API access can be restricted. Big part of why Twitter got authwalled was so that AI models can't train from it. Stack overflow added a no AI models clause to their free data dump releases (supposed to be CC licensed), they want to be paid if you use their data for AI models.

    > Agentic search

To your first point, the LLM still can’t know what it doesn’t know.

Just like you can’t google for a movie if you don’t know the genre, any scenes, or any actors in it, and AI can’t build its own context if it didn’t have good enough context already.

IMO that’s the point most agent frameworks miss. Piling on more LLM calls doesn’t fix the fundamental limitations.

TL;DR an LLM can’t magically make good context for itself.

I think you’re spot on with your second point. The big differentiators for big AI models will be data that’s not easy to google for and/or proprietary data.

Lucky they got all their data before people started caring.

This is where Microsoft has the advantage, all those Teams calls can provide context.

Yes, this is definitely a big problem.

Anyone that's done any amount of systems integration in enterprises knows this.

    "Let me talk to Lars; he should know because his team owns that system."

    "We don't have any documentation on this, but Mette should know about it because she led the project."

Difficult to impossible. Their vendors are already working on AI features, so why would they risk adding a new vendor when a vendor they've already approved will have substantially the same capabilities soon?

Context requires quadratic VRAM. It is why OpenAI hasn't even supported 200k context length yet for its 4o model.

Is there a trick that bypasses this scaling constraint while strictly preserving the attention quality? I suspect that most such tricks lead to performance loss while deep in the context.

Even if your context is a trillion tokens in length, the problem of creating that context still exists. It's still ETL and systems integration.

What acceleration?

Consider OpenAI's products as an example. GPT-3 (2020) was a massive step up in reasoning ability from GPT-2 (2019). GPT-3.5 (2022) was another massive step up. GPT-4 (2023) was a big step up, but not quite as big. GPT-4o (2024) was marginally better at reasoning, but mostly an improvement with respect to non-core functionality like images and audio. o1 (2024) is apparently somewhat better at reasoning at the cost of being much slower. But when I tried it on some puzzle-type problems I thought would be on the hard side for GPT-4o, it gave me (confidently) wrong answers every time. 'Orion' was supposed to be released as GPT-5, but was reportedly cancelled for not being good enough. o3 (2025?) did really well on one benchmark at the cost of $10k in compute, or even better at the cost of >$1m – not terribly impressive. We'll see how much better it is than o1 in practical scenarios.

To me that looks like progress is decelerating. Admittedly, OpenAI's releases have gotten more frequent and that has made the differences between each release seem less impressive. But things are decelerating even on a time basis. Where is GPT-5?

>Let's say 25 years ago you had the idea to build a product

I resemble that remark ;)

>that can now be solved more generally with LLMs

Nope, sorry, not yet.

>"Nah, don't build that business, it will eventually be solved with some new technology?"

Actually I did listen to people like that to an extent, and started my business with the express intent of continuing to develop new technologies which would be adjacent to AI when it matured. Just better than I could at my employer where it was already in progress. It took a couple years before I was financially stable enough to consider layering in a neural network, but that was 30 years ago now :\

Wasn't possible to benefit with Windows 95 type of hardware, oh well, didn't expect a miracle anyway.

Heck, it's now been a full 45 years since I first dabbled in a bit of the ML with more kilobytes of desktop memory than most people had ever seen. I figured all that memory should be used for something, like memorizing, why not? Seemed logical. Didn't take long to figure out how much megabytes would help, but they didn't exist yet. And it became apparent that you could only go so far without a specialized computer chip of some kind to replace or augment a microprocessor CPU. What kind, I really had no idea :)

I didn't say they resembled 25-year-old ideas that much anyway ;)

>We've had a lot of progress over the last 25 years; much of it in the last two.

I guess it's understandable this has been making my popcorn more enjoyable than ever ;)

> You seem to be assuming that the rapid progress in AI will suddenly stop.

And you seem to assume that it will just continue for 5 years. We've already seen the plateau start. OpenAI has tacitly acknowledged that they don't know how to make a next generation model, and have been working on stepwise iteration for almost 2 years now.

Why should we project the rapid growth of 2021–2023 5 years into the future? It seems far more reasonable to project the growth of 2023–2025, which has been fast but not earth-shattering, and then also factor in the second derivative we've seen in that time and assume that it will actually continue to slow from here.

I think you're suffering from some survivorship bias here. There are lot of technologies that don't work out.

> You seem to be assuming that the rapid progress in AI will suddenly stop.

> I think if you look at the history of compute, that is ridiculous. Making the models bigger or work more is making them smarter.

It's better to talk about actual numbers to characterise progress and measure scaling:

" By scaling I usually mean the specific empirical curve from the 2020 OAI paper. To stay on this curve requires large increases in training data of equivalent quality to what was used to derive the scaling relationships. "[^2]

"I predicted last summer: 70% chance we fall off the LLM scaling curve because of data limits, in the next step beyond GPT4.

[…]

I would say the most plausible reason is because in order to get, say, another 10x in training data, people have started to resort either to synthetic data, so training data that's actually made up by models, or to lower quality data."[^0]

“There were extraordinary returns over the last three or four years as the Scaling Laws were getting going,” Dr. Hassabis said. “But we are no longer getting the same progress.”[^1]

---

[^0]: https://x.com/hsu_steve/status/1868027803868045529

[^1]: https://x.com/hsu_steve/status/1869922066788692328

[^2]: https://x.com/hsu_steve/status/1869031399010832688

Also office jobs will be adapted to be a better fit to what AI can do, just as manufacturing jobs were adapted so that at least some tasks could be completed by robots.

Not my downvote, just the opposite but I think you can do a lot in an office already if you start early enough . . .

At one time I would have said you should be able to have an efficient office operation using regular typewriters, copiers, filing cabinets, fax machines, etc.

And then you get Office 97, zip through everything and never worry about office work again.

I was pretty extreme having a paperless office when my only product is paperwork, but I got there. And I started my office with typewriters, nice ones too.

Before long Google gets going. Wow. No-ads information superhighway, if this holds it can only get better. And that's without broadband.

But that's besides the point.

Now it might make sense for you to at least be able to run an efficient office on the equivalent of Office 97 to begin with. Then throw in the AI or let it take over and see what you get in terms of output, and in comparison. Microsoft is probably already doing this in an advanced way. I think a factor that can vary over orders of magnitude is how does the machine leverage the abilities and/or tasks of the nominal human "attendant"?

One type of situation would be where a less-capable AI could augment a defined worker more effectively than even a fully automated alternative utilizing 10x more capable AI. There's always some attendant somewhere so you don't get a zero in this equation no matter how close you come.

Could be financial effectiveness or something else, the dividing line could be a moving target for a while.

You could even go full paleo and train the AI on the typewriters and stuff just to see what happens ;)

But would you really be able to get the most out of it without the momentum of many decades of continuous improvement before capturing it at the peak of its abilities?

For me, general intelligence from a computer will be achieved when it knows when it's wrong. You may say that humans also struggle with this, and I'd agree - but I think there's a difference between general intelligence and consciousness, as you said.

> AGI in some ways

In other words, just AI, not AGI.

Also, the UX of your potential "remote workers" are vitally important! The difference between a good and a bad remote worker is almost always how good they are at communicating - both reading and understanding tickets of work to be done and how well they explain, annotate, and document the work they do.

At the end of the day, someone has to be checking the work. This is true of humans and of any potential AI agent, and the UX of that is a big deal. I can get on a call and talk through the code another engineer on my team wrote and make sure I understand it and that it's doing the right thing before we accept it. I'm sure at some point I could do that with an LLM, but the worry is that the LLM has no innate loyalty or sense of its own accuracy or honesty.

I can mostly trust that my human coworker isn't bullshitting me and any mistakes are honest mistakes that we'll learn from together for the future. That we're both in the same boat where if we write or approve malicious or flagrantly defective code, our job is on the line. An AI agent that's written bad or vulnerable code won't know it, will completely seriously assert that it did exactly what it was told, doesn't care if it gets fired, and may say completely untrue things in an attempt to justify itself.

Any AI "remote worker" is a totally different trust and interaction model. There's no real way to treat it like you would another human engineer because it has, essentially, no incentive structure at all. It doesn't care if the code works. It doesn't care if the team meets its goals. It doesn't care if I get fired. I'm not working with a peer, I'm working with an industrial machine that maybe makes my job easier.

Which is an easier way to interact with your bank? Writing a business letter, or filling out a form?

I suspect that we will still be filling out forms, because that’s a better UI for a routine business transaction. It’s easier to know what the bank needs from you if it’s laid out explicitly, and you can also review the information you gave them to make sure it’s correct.

AI could still be helpful for finding the right forms, auto-filling some fields, answering any questions you might have, and checking for common errors, but that’s only a mild improvement from what a good website already does.

And yes, it’s also helpful for the programmers writing the forms. But the bank still needs people to make sure that any new forms implement their consumer interactions correctly, that the AI assist has the right information to answer any questions, and that it’s all legal.

The amount of work going into RLHF/DPO/instruct tuning and other types of post training is because UX is very important. The bar is high and the difficulty of making a model with a good UX for a given use case is high.

A guess: Their ability to infer reality from incomplete and indirect information.

It's relatively easy to move to different cloud infrastructure (or host your own) later on down the line.

If you rely on an OpenAI LLM for your business, they can basically do whatever they want to you. Oh, prices went up 10x? What are you gonna do, train your own AI?

A LLM wrapper adds near-zero value. If I type some text into a "convert to Donald Trump style" tool, it produces the exact same output as typing it into ChatGPT following "Convert this text to Donald Trump style:" because that's what the tool actually does. Implementing ChatGPT is 99.999% of the value creation. Prepending the prompt is 0.001%. The surprising fact is that the market assigns a non-zero value to the tool anyway.

Startups that use cloud servers still write the software that goes on those servers, which is 90% of the value creation.

> We have an example. On one hand we have Tesla FSD, using throwing more and more resources an conventional AI training in the way the bitter lesson says you must do in order to progress. On the other we have Waymo using a more traditional approach. It's pretty clear which approach is failing and the other is working - and it's not going the way the bitter lesson says it should.

As I understand the article, it is going the way the bitter lesson predicts it would - the initial "more traditional" approach generates almost-workable solutions in the near term while the "bitter lesson" approach is unreliable in the near term.

Unless you think that FSD is already in the "far" term (i.e. already at the endgame), this is exactly what the article predicts happens in the near term.

Not if the generalist AI arrives after you have made your returns, which is the sentiment of the post you’re respond to.

Diversification is good advice regardless of industry / technology / niche.

Anyone that thinks this is just years away is utterly ignoring human history, nature, and relationship with technology. My own view is that this will never be achieved, and it's not even just about the tech.

Let's imagine for a moment that this is even achieved. Then, there is still complex engineering required in the world: to maintain and continually improve the AI engines and their interfaces. Unless you want to say that, past some point, the AI will be self-improving without any human input whatsoever. Unless the AI can read our minds, I'm not sure it can continue to serve human interests without human input.

But, never mind, we will never get there. At this very moment, tech is capable of so much more, but most sites I visit have bad UI, are bloated downloading and executing massive amounts of JS, riddled with annoying ads that serve no real useful purpose to society, and riddled with bugs. Even as an engineer, I really struggle to find any good no-code tools to create anything truly sophisticated without digging into hard-core code. Heck, they are now talking about adding more HTTP methods to HTML forms.

A human is not a blank slate. There's millennia of evolutionary history that goes into making a brain adapted and capable of learning from its environment.

> A human does not need billions of driving hours to learn how to drive competently.

But humans DO need ~16 years of growth and development "to learn how to drive competently" and then will also know how to ride a bycycle, mow grass, build shelves, cook pizza, use a smart phone, ...! There's a lesson in that somewhere ....

I think there's a more fundamental problem at play here: what seems to work in 'AI', search, is made better by throwing more data into more compute. You then store the results in a model, that amounts to pre-computed solutions waiting for a problem. Interacting with the model is then asking questions and getting answers that hopefully fit your needs.

So, what we're doing on the whole seems to be a lot of coding and decoding, hoping that the data used in training can be adequately mapped to the problem domain realities. That would mean that the model you end up with is somehow a valid representation of some form of knowledge about the problem domain. Trouble is, more text won't yield higher and higher resolution of some representation of the problem domain. After some point, you start to introduce noise.

Yeah well. That was a bad analogy, and everyone I know who used to say that, admits error.