The Darwin Gödel Machine: AI that improves itself by rewriting its own code

> I don’t think they can yet self improve exponentially without human intuition yet

I agree: if they could, they would be doing it already.

Case in point: one of the first things done once ChatGPT started getting popular was "auto-gpt"; roughly, let it loose and see what happens.

The same thing will happen to any accessible model in the future. Someone, somewhere will ask it to self-improve/make as much money as possible, with as little leashes as possible. Maybe even the labs themselves do that, as part of their post-training ops for new models.

Therefore, we can assume that if the existing models _could_ be doing that, they _would_ be doing that.

That doesn't say anything about new models released 6 months or 2 years from now.

Note that this isn't improving the LLM itself, but the software glue around it (i.e. agentic loops, tools, etc). The fact that using the same LLM got ~20% increase on the aider leaderboard speaks more about aider as a collection of software glue, than it does about the model.

I do wonder though if big labs are running this with model training episodes as well.

Don't take this the wrong way, your opinion is also vibes.

Let's ground that a bit.

Have a look at ARC AGI 1 challenge/benchmark. Solve a problem or two yourself. Know that ARC AGI 1 is practically solved by a few LLMs as of Q1 2025.

Then have a look at the ARC AGI 2 challenge. Solve a problem or two yourself. Note that as of today, it is unsolved by LLMs.

Then observe that the "difficulty" of ARC AGI 1 and 2 for a human are relatively the same but challenge 2 is much harder for LLMs than 1.

ARC AGI 2 is going to be solved *within* 12 months (my bet is on 6 months). If it's not, I'll never post about AI on HN again.

There's only one problem to solve, i.e. "how to make LLMs truly see like humans do". Right now, any vision based features that the models exhibit comes from maximizing the use of engineering (i.e. applying CNNs on image slices, chunks, maybe zooming and applying ocr, vector search etc), it isn't vision like ours and isn't a native feature for these models.

Once that's solved, then LLMs or new Algo will be able to use a computer perfectly by feeding it screen capture. End of white collar jobs 2-5 years after (as we know it).

Edit - added "(as we know it)". And fixed missing word.

The wall is training data. An AI can't produce its own training data because an AI can't be smarter than its own training data. This is a well known regression problem and one I personally believe is not solvable. (A softer assertion would be: it's not solvable with current technology.)

I agree , it might incrementally optimize itself very well, but i think for now at least anything super innovative will still come from a human that can think beyond a few steps. There are surely far better possible architectures, training methods etc that would initially lead to worse performance if approached stepwise.

That would be something. When a AI/LLM can create new axioms or laws that have not discovered by humanity.

what is there to improve? the transformer architecture is extremely simple. you gonna add another kv layer? you gonna tweak the nonlinearities? you gonna add 1 to one of the dimensions? you gonna inject a weird layer (which could have been in the weights anyways due to kolmogorov theorem)?

realistically the best you could do is evolve the prompt. maybe you could change input data preprocessing?

anyways the idea of current llm architectures self-improving via its own code seems silly as there are surprisingly few knobs to turn, and it's ~super expensive to train.

as a side note it's impressive how resistant the current architecture is to incremental RL away from results, since if even one "undesired input" result is multiple tokens, the coupling between the tokens is difficult to disentangle. (how do you separate jinping from jin-gitaxias for example)

an LLM can't learn without adding new data and a training run. so it's impossible for it to "self improve" by itself.

I'm not sure how much an agent could do though given the right tools. access to a task mgt system, test tracker. robust requirements/use cases.

My favorite was always the HumanEval dataset.

  Problem: 
    1) we want to train on GitHub repos
    2) most datasets are spoiled. Training on GitHub would definitely spoil

  Solution:
    Hand write new problems!!!
    ... leetcode style ....
    ... and we'll check if it passes test

  Example:
    What's the decimal part of this float?

Sure in all of GitHub we can filter such code out by ngram!

Maybe my favorite part is that it has 60 authors and became the de facto benchmark for awhile

Model weights are code, for a dive into that see [0]. That shows how to encode Boolean logic using NAND gates in an MLP.

The expressivity is there, the only question is how to encode useful functions into those weights, especially when we don’t know how to write those functions by hand.

[0] http://neuralnetworksanddeeplearning.com/chap1.html

If it can generate the model (from training data) then presumably that'd be fine, but the iteration time would be huge and expensive enough to be currently impractical.

Or yeah if it can modify its own weights sensibly, which feels ... impossible really.

Now here's the tricky part:

What's the difference?

Give it some serious thought. Challenge whichever answer you come up with. I guarantee this will be trickier than you think

It’s more like a nightly training, isn’t it? IIUC the human brain learns from its experiences while it’s asleep, so it might be kind of like taking things out of context windows and fine tuning on them every night.

Correct and working on it. You can take the approach of mixed experts and train the network in chunks that share known interfaces over which they communicate results. These chunks can be trained on their own, but you cannot have a set training set here.

Then if you go further and alter the architecture by introducing clean category theory morphisms and build from there you can have a dynamic network - but you will still have to retrain this network every time you change the structure.

You can spin this further and know the need for a real-world training set and a loss function that will have to competete against other networks. In the end a human brain is already best at this and embodied in the real world.

What i want to add here is that our neurons not take in weights - they also fire depending on whether one input comes after another or before and differs down to the nanoseconds here - unmatched in IT and ofc heaps more efficient.

I still would say its possible though and currently work on 4D lifeforms built on dynamic compute graphs that can do this in a set virtual environment.

So this is pretty awesome stuff, but its a long fetch from anything we do right now.

Reward hacking is a well known and tracked problem at frontier labs - Claude 4’s system card reports on it for instance. It’s not surprising that a framework built on current llms would have reward hacking tendencies.

For this part of the stack the interesting question to me is how to identify and mitigate.

> This demonstrates a technique whereby a smaller/older/cheaper model has been used to improve the output of a larger model. This is backwards (as far as I understand). The current SOTA technique typically sees enormous/expensive models training smaller cheaper models.

There are two separate aspects here. In this paper they improve the software around the model, not the model itself. What they're saying is that the software improvements carried over to other models, so it wasn't just optimising around model-specific quirks.

What you're describing with training large LLMs first is usually called "distillation" and it works on training the smaller LLM to match the entire distribution of tokens at once (hence it's faster in practice).

I think it's different from improving the model weights themselves, like the distillation examples you are mentioning. It's that changes to the "harness" or code running around the llm calls (which is what this is editing), persist or generalize to wrapping more powerful llms. That means they aren't all wasted when a more powerful llm comes along that the harness wasn't tuned to use.

I was a bigger fan of the certain doom in 2025, and I think the AI 2030 movement will have better design sense and storytelling. But really I haven’t seen anything that really has the oomph and fire of Tipper Gore’s crusade against youth music.

We need more showmanship, more dramatic catastrophizing. I feel like our current crop of doomers isn’t quite shameless enough to be really entertaining.

People should understand that the reason this seemingly fan-fict blog post gets so much traction is because of lead author's August 2021 "fan-fict" blog post, "What 2026 Looks Like":

https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-...

This reads like an advertisement for OpenBrain and doesn't seem grounded in reality.

Checked out when it turned into bad geopolitics fiction.

So this is what the crowd of people who write SCP articles with over 1000 upvotes does in their professional life?

[dead]