Wasting Inferences with Aider

> One of the (many) differences between junior developers and LLM assistance is that humans can learn from their mistakes

One would think so, but I've had some developers repeat the same mistake a hundred times, where eventually they admit they just keep forgetting it.

The frustration you feel when telling a human for the Xth time that we do not allow yoda-conditions in our codebase is incredibly similar to when an AI does something wrong.

> often

Often being about 30% of the time in my experience

I think this is an interesting idea, but I also somewhat suspect you've replaced a tedious problem with a harder, more tedious problem.

Take your idea further. Now I've got 100 agents, and 100 PRs, and some small percentage of them are decent. The task went from "implement a feature" to "review 100 PRs and select the best one".

Even assuming you can ditch 50 percent right off the bat as trash... Reviewing 50 potentially buggy implementations of a feature and selecting the best genuinely sounds worse than just writing the solution.

Worse... If you haven't solved the problem before anyways, you're woefully unqualified as a reviewer.

> It may not be as stupid as it sounds.

It is.

> However, in any probabilistic system, you can run a function k times and you’ll get an output distribution that is meaningful if k is high enough.

This is the underlying flaw in this approach. Attempting to use probabilistic algorithms to produce a singular verifiably correct result requires an external agent to select what is correct in the output of "k times" invocations. This is a person capable of making said determination.

> The thing with LLMs is they scale in a way that actually allows this to be possible, in a way that scaling with humans can’t.

For the "k times" generation of text part, sure. Not for the determination of which one within k, if any, are acceptable for the problem at hand.

EDIT: clarified "produce a verifiably correct result" to be "produce a singular verifiably correct result"

> like the monkeys and Shakespeare, there probably a limit to the value it can offer; but it’s not totally meaningless to try it.

Whenever someone uses this analogy, a question never addressed is:

  Assuming sufficient monkeys, typewriters, and time, how
  would anyone know if a Shakespearean work was produced
  unless one reviewed *all* output?

> taking some kind of general solution from the distribution

My instinct is that this should be the temperature 0K response (no randomness).

Three is high enough, in my eyes. Two might be. Remember that we don't care about any but the best solution. With one sample you've only got one 50/50 shot to get above the median. With three, the odds of the best of the three being above the median is 87.5%.

Of course picking the median as the random crap boundary is entirely arbitrary, but it'll do until there's a justification for a better number.

correct.

Continual feedback means continual training. No way around it. So you’d have to scope down the functional unit to a fairly small lora in order to get reasonable re-training costs here.

> This is a huge opportunity, maybe the next big breakthrough in AI when someone figures out how to solve it

I am not saying I solved it, but I believe we are going to experience a paradigm shift in how we program and teach and for some, they are really going to hate it. With AI, we can now easily capture the thought process for how we solve problems, but there is a catch. For this to work, senior developers will need to come to terms that their value is not in writing code, but solving problems.

I would say 98% of my code is now AI generated and I have 0% fear that it will make me dumber. I will 100% become less proficient in writing code, but my problem solving skills will not go away and will only get sharper. In the example below, 100% of the code/documentation was AI generated, but I still needed to guide Gemini 2.5 Pro

https://app.gitsense.com/?chat=c35f87c5-5b61-4cab-873b-a3988...

After reviewing the code, it was clear what the problem was and since I didn't want to waste token and time, I literally suggested the implementation and told it to not generate any code, but asks it to explain the problem and the solution, as shown below.

> The bug is still there. Why does it not use states to to track the start @@ and end @@? If you encounter @@ , you can do an if else on the line by asking if the line ends with @@. If so, you can change the state to expect replacement start delimiter. If it does not end with @@ you can set the state to expect line to end with @@ and not start with @@. Do you understand? Do not generate any code yet.

How I see things evolving over time is, senior developers will start to code less and less and the role for junior developers will not only be to code but to review conversations. As we add new features and fix bugs, we will start to link to conversations that Junior developers can learn from. The Dooms day scenario is obviously, with enough conversations, we may reach the point where AI can solve most problems one shot.

Full Disclosure: This is my tool

We feed it conventions that are automatically loaded for every LLM task, do that the LLM adheres to coding style, comment style, common project tooling and architecture etc.

These systems don't do online learning, but that doesn't mean you can spoon feed them what they should know and mutate that knowledge over time.

Are you sure about "all"? Because I mentioned not only env deployment, but also functional issue reproduction using UI/API, which is also require necessary pre-setup.

Automated tests partially solve the case, but in real world no one writes tests blindly. It's always manual work, and when the failing trajectory is clear - the test is written.

Theoretically agent can interact with UI or API. But it requires deep project understanding, gathered from code, documentation, git history, tickets, slack. And obtaining this context, building an easily accessible knowledge base and puring only necessary parts into the agent context - is still a not solved task.

If your CI/CD process was able to fully verify a fix then it would have stopped the bug from making it to production the first time around and the Jira ticket which was handed to multiple LLMs never would have existed.

Look at this 18 years old Django ticket: https://code.djangoproject.com/ticket/4140

It was impossible to fix, but it required some experiments and deep research about very specific behaviors.

Or this ticket: https://code.djangoproject.com/ticket/35289

Author proposed one-line solution, but the following discussion includes analysis of RFC, potential negative outcomes, different ways to fix it.

And without deep understanding of the project - it's not clear how to fix it properly, without damage to backward compatibility and neighbor functionality.

Also such a fix must be properly tested manually, because even well designed autotests are not 100% match the actual flow.

You can explore other open and closed issues and corresponding discussions. And this is the complexity level of real software, not pet projects or simple apps.

I guess that existing attention mechanism is the fundamental blocker, because it barely able to process all the context required for a fix.

And feature requests a much, much more complex.

Have you tried building agents? They will go from PhD level smart to making mistakes a middle schooler would find obvious, even on models like gemini-2.5 and o1-pro. It's almost like building a sandcastle where once you get a prompt working you become afraid to make any changes because something else will break.

The fundamental blocker: Context

you can just even tell cursor to use any cli tools you use normally in your development, like git, gh, railway, vercel, node debugging, etc etc

Correct! Over at https://ghuntley.com/mcp I propose that each company develops their own tools for their particular codebase that shapes LLM actions on how to work with their codebase.

So... I know that people frame these sorts of things as if it's some kind of quantization conspiracy, but as someone who started using Claude Code the _moment_ that it came out, it felt particularly strong. Then, it feels like they... tweaked something, whether in CC or Sonnet 3.7 and it went a little downhill. It's still very impressive, but something was lost.

I've found Gemini 2.5 Pro to be extremely impressive and much more able to run in an extended fashion by itself, although I've found very high variability in how well 'agent mode' works between different editors. Cursor has been very very weak in this regard for me, with Windsurf working a little better. Claude Code is excellent, but at the moment does feel let down by the model.

I've been using Aider with Gemini 2.5 Pro and found that it's very much able to 'just go' by itself. I shipped a mode for Aider that lets it do so (sibling comment here) and I've had it do some huge things that run for an hour or more, but assuredly it does get stuck and act stupidly on other tasks as well.

My point, more than anything, is that... I'd try different editors and different (stronger) models and see - and that small tweaks to prompt and tooling are making a big difference to these tools' effectiveness right now. Also, different models seem to excel at different problems, so switching models is often a good choice.

Right, but it seems like that would just generate three PRs I don’t want to review, given the likelihood the agent went into the weeds without someone supervising it.

It does for me, yes -- models seem to be pretty capable of adhering to the tool call format, which is really all that they 'need' in order to do a good job.

I'm still tweaking the prompts (and I've introduced a new, tool-call based edit format as a primary replacement to Aider's usual SEARCH/REPLACE, which is both easier and harder for LLMs to use - but it allows them to better express e.g. 'change the name of this function').

So... if you have any trouble with it, I would adjust the prompts (in `navigator_prompts.py` and `navigator_legacy_prompts.py` for non-tool-based editing). In particular when I adopted more 'terseness and proactively stop' prompting, weaker LLMs started stopping prematurely more often. It's helpful for powerful thinking models (like Sonnet and Gemini 2.5 Pro), but for smaller models I might need to provide an extra set of prompts that let them roam more.

It's funny you say this! I was adding a tool just earlier (that I haven't yet pushed) that allows the model to... switch model.

Aider can also have multiple models active at any time (the architect, editor and weak model is the standard set) and use them for different aspects. I could definitely imagine switching one model whilst leaving another active.

So yes, this definitely seems feasible.

Aider had a fairly coherent answer to this question, I think: https://gist.github.com/tekacs/75a0e3604bc10ea88f9df9a909b5d...

This was navigator mode + Gemini 2.5 Pro's attempt at implementing it, based only on pasting in your comment:

https://asciinema.org/a/EKhno9vQlqk9VkYizIxsY8mIr

https://github.com/tekacs/aider/commit/6b8b76375a9b43f9db785...

I think it did a fairly good job! It took just a couple of minutes and it effectively just switches the main model based on recent input, but I don’t doubt that this could become really robust if I had poked or prompted it further with preferences, ideas, beliefs and pushback! I imagine that you could very quickly get it there if you wished.

It's definitely not showing off the most here, because it's almost all direct-coding, very similar to ordinary Aider. :)

Wonder if you could have the reviewer characterize any mistakes and feed those back into the coding prompt: “be sure to… be sure not to…”

You can have a larger model split things out into more manageable steps and create new tickets - marked as blocked or not on each other, then have the whole thing run.

Technically that's known as "LLM-as-judge" and it's all over the literature. The intuition would be that the capability to choose between two candidates doesn't exactly overlap with the ability to generate either one of them from scratch. It's a bit like how (half of) generative adversarial networks work.

s/AI/tech

Agents could generate more PRs in a weekend than my team could code review in a month.

Initially we can absolutely just review them like any other PR, but at some point code review will be the bottleneck.

I suspect this will indeed be part of it, but it won't work with today's AIs on today's codebases.

Models will improve, but also I predict code style and architecture will evolve towards something easier for machine review.