Does the Bitter Lesson Have Limits?
65 comments
·August 1, 2025sorenjan
bsoles
But I have also seen people trying to use deep networks to identify rotating machinery faults, like bearings, from raw accelerometer data collected a high frequencies like 40 kHz. Whereas the spectrum of the data from running FFT on the signal contains fault information much more obviously and clearly.
Throwing a deep network on a problem without some physical insight into the problem has also its disadvantages it seems.
jvanderbot
Yeah, we're shouting into the wind here. I have had people tell me directly that my ideas from old school state estimation were irrelevant in the era of deep learning. They may produce (in this case) worse results, but the long game I'm assured is superior.
The specific scenario was estimating the orientation of a stationary semi trailer. An objectively measurable number and it was consistently off by 30 deg, yet I was the jerk for suggesting we move from end to end DL to trad Bar Shalom techniques.
That scene isn't for me anymore.
physicsguy
I’m working on this sort of thing right now in a SaaS product that previously didn’t have support for vibration data. One competitor is all ML-d up to the hilt but customers don’t like the black box and keep finding it gives false positives with no explanation. I think one problem is those buying not understanding problem; they just want to plug in a sensor and insights to happen, but without information about the machine that’s never going to be able to provide useful insights
bsoles
There are also extremely misleading research articles out there promising good results with deep networks in the area of anomaly detection, without adequate comparison with more classical techniques.
This well-known critical paper shows examples of AI articles/techniques applied to popular datasets with good-looking results. But, it also demonstrates that, literally, a single line of MATLAB code can outperform some of these techniques: https://arxiv.org/pdf/2009.13807
ssivark
> an autonomous drone that beat the best human FPV pilots
Doesn’t any such claim come with huge caveats — pre specified track/course, no random objects flying between, etc…? ie. train & test distributions are ensured same by ensuring test time can never be more complicated than training data.
Also presumably better sensing than raw visual input.
logicchains
>It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives
It's similar with options pricing. The most sophisticated models like multivariate stochastic volatility are computationally expensive to approximate with classical approaches (and have no closed form solution), so just training a small NN on the output of a vast number of simulations of the underlying processes ends up producing a more efficient model than traditional approaches. Same with stuff like trinomial trees.
cactusfrog
This is really interesting. I think force fields in molecular dynamics have underwent a similar NN revolution. You train your NN on the output of expensive calculations to replace the expensive function with a cheap one. Could you train a small language model with a big one?
lossolo
> Could you train a small language model with a big one?
Yes, it's called distillation.
William_BB
Interesting. Are these models the SOTA in the options trading industry (e.g. MM) nowadays?
lisper
Por que no los dos?
Thirty-five years ago they gave me a Ph.D. basically for pointing out that the controversy du jour -- reactive vs deliberative control for autonomous robots -- was not a dichotomy. You could have the best of both worlds by combining a reactive system with a deliberative one. The reactive system interfaced directly to the hardware on one end and provided essentially a high-level API on the other end that provided primitives like "go that way". It's a little bit more complicated than that because it turns out you need a glue layer in the middle, but the point is: you don't have to choose. The Bitter Lesson is simply a corollary of Ron's First Law: all extreme positions are wrong. So reactive control by itself has limits, and deliberative control by itself has limits. But put the two together (and add some pretty snazzy image processing) and the result is Waymo.
So it was no surprise to me that Stockfish, with its similar approach of combining deliberative search with a small NN computing its quality metric blows everything else out of the water. It has been obvious (at least to me) that this is the right approach for decades now.
I'm actually pretty terrified of the results when the mainstream AI companies finally rediscover this. The capabilities of LLMs are already pretty impressive on their own. If they can get a Stockfish-level boost by combining them with a simple search algorithm the result may very well be the GAI that the rationalist community has been sounding the alarm over for the last 20 years.
grubbypaw
I was not at all a fan of "The Bitter Lesson versus The Garbage Can", but this misses the same thing that it missed.
The Bitter Lesson is from the perspective of how to spend your entire career. It is correct over the course of a very long time, and bakes in Moore's Law.
The Bitter Lesson is true because general methods capture these assumed hardware gains that specific methods may not. It was never meant for contrasting methods at a specific moment in time. At a specific moment in time you're just describing Explore vs Exploit.
schmidtleonard
Right, and if you spot a job that needs doing and can be done by a specialized model, waving your hands about general purpose scale-leveraging models eventually overtaking specialized models has not historically been a winning approach.
Except in the last year or two, which is why people are citing it a lot :)
hinkley
Probably because this is how bubbles happen.
anp
I think there might be interesting time scales in between “now” and “my entire career” to which the bitter lesson may or may not apply. As an outsider to ML I have questions about the longevity of any given “context engineering” approach in light of the bitter lesson.
energy123
The bitter lesson becomes more true over time, because inductive bias becomes less useful over time. Case in point: PCA/hand engineering -> CNN -> ViT.
itkovian_
I don’t think people understand the point sutton was making; he’s saying that general, simple systems that get better with scale tend to outperform hand engineered systems that don’t. It’s a kind of subtle point that’s implicitly saying hand engineering inhibits scale because it inhibits generality. He is not saying anything about the rate, doesn’t claim llms/gd are the best system, in fact I’d guess he thinks there’s likely an even more general approach that would be better. It’s comparing two classes of approaches not commenting on the merits of particular systems.
joe_the_user
It occurs to me that the bitter lesson is so often repeated because it involves a slippery slope or moot-and-castle argument. IE, the meaning people assign to the bitter lesson ranges between all the following:
General-purpose-algorithms-that-scale will beat algorithms that aren't those
The most simple general purpose, scaling algorithm will win, at least over time
Neural networks will win
LLMs will reach AGI with just more resources
arrowsmith
motte and bailey*
eldenring
Yep this article is self centered and perfectly represents the type of ego Sutton was referencing. Maybe in a year or two general methods will improve the author's workflow significantly once again (eg. better models) and they would still add a bit of human logic on top and claim victory.
visarga
The point about training data stands. We usually only think of scaling compute, but we need to scale data as well, maybe even faster than compute. But we exhausted the source of high quality organic text, and it doesn't grow exponentially fast.
I think at the moment the best source of data is the chat log, with 1B users and over 1T daily tokens over all LLMs. These chat logs are at the intersection of human interests and LLM execution errors, they are on-policy for the model, right what they need to improve the next iteration.
Animats
The question is when price/performance hits financial limits. That point may be close, if not already passed.
Interestingly, this hasn't happened for wafer fabs. A modern wafer fab costs US$1bn to US$3bn, and there is talk of US$20bn wafer fabs. Around the year 2000, those would have been un-financeable. It was expected that fab cost was going to be a constraint on feature size. That didn't happen.
For years, it was thought that the ASML approach to extreme UV was going to cost too much. It's a horrible hack, shooting off droplets of tin to be vaporized by lasers just to generate soft X-rays. Industry people were hoping for small synchrotrons or X-ray lasers or E-beam machines or something sane. But none of those worked out. Progress went on by making a fundamentally awful process work commercially, at insane cost.
schmidtleonard
Fundamentally awful but spiritually delightful.
azeirah
Sometimes awful is the best we have, we don't have anything that performs at a similar level to EUV machines by ASML but are much simpler or tenable than what we have right now, right?
Perhaps we will find something better in the future, but for now awful is the best we got for the cutting edge.
Also, when is cutting edge not the worst it's ever been?
xpe
> The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.
Be careful when anyone, even a giant in the field such as Sutton, posits a sweeping claim like this.
My take? Sutton's "bitter lesson" is rather vague and unspecified (i.e. hard to pin down and test) for at least two reasons:
1. The word "ultimately" is squishy, when you think about it. When has enough time passed to make the assessment? At what point can we say e.g. "Problem X has a most effective solution"?
2. What do we mean by "most effective"? There is a lot of variation, including but not limited to (a) some performance metric; (b) data efficiency; (c) flexibility / adaptability across different domains; and (d) energy efficiency.
I'm a big fan of Sutton's work. I've read his RL book cover-to-cover and got to meet him briefly. But, to me, the bitter lesson (as articulated in Sutton's original post) is not even wrong. It is sufficiently open-ended that many of us will disagree about what the lesson is, even before we can get to the empirical questions of "First, has it happened in domain D at time T? Second, is it 'settled' now, or might things change?"
softwaredoug
In my experience, there's an opposing "bitter lesson" when trying to make incremental, tactical progress in user-facing AI / ML applications: _you're not a researcher_. Stay to tried-and-true, boring ML methods that have been proven at scale and then add human knowledge and rules to make it all work.
Then, as the article mentions, some new fundamental shift happens, and practitioners need to jump over to a completely new way of working. Monkeypatching to make it all work. Rinse repeat.
PaulHoule
One odd thing is that progress in SAT/SMT solvers has been almost as good as progress in neural networks from the 1970s to the present. There was a time I was really interested in production rules and expert system shells and systems in the early 1980s often didn't even use RETE and didn't have hash indexes so of course a rule base of 10,000 looked unmanageable, by 2015 you could have a million rules in Drools and it worked just fine.
bmc7505
The difference is that SAT/SMT solvers have primarily relied on single-threaded algorithmic improvements [1] and unlike neural networks, we have not [yet] discovered a uniformly effective strategy for leveraging additional computation to accelerate wall-clock runtime. [2]
PaulHoule
RETE family algorithms did turn out to be somewhat parallelizable, enough to get a real speed-up on ordinary multicore CPUs. There was an idea in the 1980s that symbolic AI would be massively parallelizable that turned out to be a disappointment.
https://en.wikipedia.org/wiki/Fifth_Generation_Computer_Syst...
jamesblonde
I see elements of the bitter lesson in arguments about context window size and RAG. The argument is about retrieval being the equivalent of compute/search. Just improve them, to hell with all else.
However, retrieval is not just google search. Primary key lookups in my db are also retrieval. As are vector index queries or BM25 free text search queries. It's not a general purpose area like compute/search. In summary, i don't think that RAG is dead. Context engineering is just like feature engineering - transform the swamp of data into a structured signal that is easy for in-context learning to learn.
The corollory of all this is it's not just about scaling up agents - giving them more LLMs and more data via MCP. The bitter lesson doesn't apply to agents yet.
thrawa8387336
This brings about a good point:
How much of the recent bitter lesson peddling is done by compute salesmen?
How much of it is done by people who can buy a lot of compute?
Deepseek was scandalous for a reason.
pu_pe
I'm not so sure Stockfish is a good example. The fact it can run on an Iphone is due to Moore's law, which follows the same pattern. And Deepmind briefly taking its throne was a very good example of the Bitter Lesson.
benreesman
When The Bitter Lesson essay came out it was a bunch of important things: addressing an audience of serious practitioners, contrarian and challenging entrenched dogma, written without any serious reputational or (especially) financial stake in the outcome. It needed saying and it was important.
But its become a lazy crutch for a bunch of people who meet none of those criteria and perverted into a statement more along the lines of "LLMs trained on NVIDIA cards by one of a handful of US companies are guaranteed to outperform every other approach from here to the Singularity".
Nope. Not at all guaranteed, and at the moment? Not even looking likely.
It will have other stuff in it. Maybe that's prediction in representation space like JEPA, maybe its MCTS like Alpha*, maybe its some totally new thing.
And maybe it happens in Hangzhou.
The last time I was reminded of the bitter lesson was when I read about Guidance & Control Networks, after seeing them used in an autonomous drone that beat the best human FPV pilots [0]. Basically it's using a small MLP (Multi Layer Perceptron) on the order of 200 parameters, and using the drone's state as input and controlling the motors directly with the output. We have all kinds of fancy control theory like MCP (Model Predictive Control), but it turns out that the best solution might be to train a relatively tiny NN using a mix of simulation and collected sensor data instead. It's not better because of huge computation resources, it's actually more computationally efficient than some classic alternatives, but it is more general.
[0] https://www.tudelft.nl/en/2025/lr/autonomous-drone-from-tu-d...
https://www.nature.com/articles/s41586-023-06419-4
https://arxiv.org/abs/2305.13078
https://arxiv.org/abs/2305.02705