Gemini 2.5 Pro Preview

I can't point to any evidence. Also I can't think of what direct evidence I could present that would be convincing, short of an actual demonstration? I would like to try to justify my intuition though:

Seems like the key question is: should we expect AI programming performance to scale well as more compute and specialised training is thrown at it? I don't see why not, it seems an almost ideal problem domain?

* Short and direct feedback loops

* Relatively easy to "ground" the LLM by running code

* Self-play / RL should be possible (it seems likely that you could also optimise for aesthetics of solutions based on common human preferences)

* Obvious economic value (based on the multi-billion dollar valuations of vscode forks)

All these things point to programming being "solved" much sooner than say, chemistry.

> I code with multiple LLMs every day and build products that use LLM tech under the hood. I dont think we're anywhere near LLMs being good at code design.

I too use multiple LLMs every day to help with my development work. And I agree with this statement. But, I also recognize that just when we think that LLMs are hitting a ceiling, they turn around and surprise us. A lot of progress is being made on the LLMs, but also on tools like code editors. A very large number of very smart people are focused on this front and a lot of resources are being directed here.

If the question is:

Will the LLMs get good at code design in 5 years?

I think the answer is:

Very likely.

I think we will still need software devs, but not as many as we do today.

You're using them in reverse. They are perfect for generating code according to your architectural and code design templete. Relying on them for architectural design is like picking your nose with a pair of scissors - yeah technically doable, but one slip and it all goes to hell.

The software tool takes a higher-level input to produce the executable.

I'm waiting for LLMs to integrate directly into programming languages.

The discussions sound a bit like the early days of when compilers started coming out, and people had been using direct assembler before. And then decades after, when people complained about compiler bugs and poor optimizers.

We're talking about predicting the future, so we can only extrapolate.

Seeing the evidence you're thinking of would mean that LLMs will have solved software development by next month.

I run a software development company with dozens of staff across multiple countries. Gemini has us to the point where we can actually stop hiring for certain roles and staff have been informed they must make use of these tools or they are surplus to requirements. At the current rate of improvement I believe we will be operating on far less staff in 2 years time.

[dead]

https://chatgpt.com/c/681aa95f-fa80-8009-84db-79febce49562

it becomes a question of how much you believe it's all just training data, and how much you believe the LLM's got pieces that are composable. I've given the question on the link as an interview questions and had humans been unable to give as through an answer (which I chose to believe is due to specialization on elsewhere in the stack). So we're already at a place where some human software development abilities have been eclipsed on some questions. So then even if the underlying algorithms don't improve, and they just ingest more training data, then it doesn't seem like a total guess as to what part of the S-curve we're on - the number of questions for software development that LLMs are able to successfully answer will continue to increase.

I had a coworker making very similar claims recently - one of the more AI-positive engineers on my team (a big part of my department's job is assessing new/novel tech for real-world value vs just hype). I was stunned when I actually saw the output of this process, which was a multi-page report describing the architecture of an internal system that arguably needed an overhaul. I try to keep an open mind, but this report was full of factual mistakes, misunderstandings, and when it did manage to accurately describe aspects of this system's design/architecture, it made only the most surface-level comments about boilerplate code and common idioms, without displaying any understanding of the actual architecture or implications of the decisions being made. Not only this coworker but several other more junior engineers on my team proclaimed this to be an example of the amazing advancement of AI ... which made me realize that the people claiming that LLMs have some superhuman ability to understand and design computer systems are those who have never really understood it themselves. In many cases these are people who have built their careers on copying and pasting code snippets from stack overflow, etc., and now find LLMs impressive because they're a quicker and easier way to do the same.

I would argue that what LLMs are capable of doing right now is already pretty extraordinary, and would fulfil your extraordinary evidence request. To turn it on its head - given the rather astonishing success of the recent LLM training approaches, what evidence do you have that these models are going to plateau short of your own abilities?

Beating humans isnt really what matters. Its enabling developers to design who cant.

Last month I had a staff member design and build a distributed system that would be far beyond their capabilities without AI assistance. As a business owner this allows me to reduce the dependency and power of the senior devs.

We were all crazy hyped when NVIDIA demoed end-to-end self driving, weren't we? First order derivatives of a hype cycle curve at lower X values is always extremely large but it's not so useful. At large X it's obviously obvious. It's always had been that way.

Trends would dictate that this will keep scaling and surpass each goalpost year by year.

I recently asked o4-mini-high for a system design of something moderately complicated and provided only about 4 paragraphs of prompt for what I wanted. I thought the design was very good, as was the Common Lisp code it wrote when I asked it to implement the design; one caveat though: it did a much better job implementing the design in Python than Common Lisp (where I had to correct the generated code).

My friend, we are living in a world of exponential increase of AI capability, at least for the last few years - who knows what the future will bring!

[dead]

AlphaGo.

I think an analogy that is helpful is that of a woodworker. Automation just allowed them to do more things at in less time.

Power saws really reduced time, lathes even more so. Power drills changed drilling immensely, and even nail guns are used on roofing project s because manual is way too slow.

All the jobs still exist, but their tools are way more capable.

> Informing product roadmaps, balancing tradeoffs, understanding relationships between teams, prioritizing between separate tasks, pushing back on tech debt, responding to incidents, it’s a feature and not a bug, …

Ask yourself how many of these things still matter if you can tell an AI to tweak something and it can rewrite your entire codebase in a few minutes. Why would you have to prioritize, just tell the AI everything you have to change and it will do it all at once. Why would you have tech debt, that's something that accumulates because humans can only make changes on a limited scope at a mostly fixed rate. LLMs can already respond to feedback about bugs, features and incidents, and can even take advice on balancing tradeoffs.

Many of the things you describe are organizational principles designed to compensate for human limitations.

Software engineering (and most professions) also have something that LLMs can't have: an ability to genuinely feel bad. I think [1] it's hugely important and is an irreducible advantage that most engineering-adjacent people ignore for mostly cultural reasons.

[1]: https://dgroshev.com/blog/feel-bad/

The way I see it:

* The world is increasingly ran on computers.

* Software/Computer Engineers are the only people who actually truly know how computers work.

Thus it seems to me highly unlikely that we won't have a job.

What that job entails I do not know. Programming like we do today might not be something that we spend a considerable amount of time doing in the future. Just like most people today don't spend much time handing punched-cards or replacing vacuum tubes. But there will still be other work to do, I don't doubt that.

In the world where idle speculation can be passed off as established future facts, i.e., this one I guess.

Proof by negation, I guess?

If someone were to claim: no computer will ever be able to beat humans in code design, would you agree with that? If the answer is "no", then there's your proof.

In the delusional startup world.

> * "my coworkers will just ruin it"

This turns out to be a big issue. I read everything about software design I could get my hands on in years, but then at an actual large company it turned out to not help, because I'd never read anything about how to get others to follow the advice in my head from all that reading.

It’s always so aggressive too. What fools we are for trying to write maintainable code when it’s so obviously impossible.

I think they are pointing out that the advantage humans have has been chipped away little by little and computers winning at coding is inevitable on some timeline. They are also suggesting that perhaps the GP is being defensive.

I agree that there’s a lot of ambiguity and tacit information that goes into building code. I wonder if that won’t change directly as a result of wanting to get more value out of agentic AI coders.

> Sure, if you write down a summary of each and every meeting you've attended for the past 12 months, as well as attach your entire company confluence, into the prompt, perhaps then the LLM can design the right architecture. But is that realistic?

I think it is definitely realistic. Zoom and Confluence already have AI integrations. To me it doesn’t seem long before these tools and more become more deeply MCPified, with their data and interfaces made available to the next generation of AI coders. “I’m going to implement function X with this specific approach based on your conversation with Bob last week.”

It strikes me that remote first companies may be at an advantage here as they’re already likely to have written artifacts of decisions and conversations, which can then provide more context to AI assistants.

Both AWS and Azure docs’ built in models have been absolutely useless.

We can rewind the clock 10 years and I can substitute "vibe coding" for VBA/Excel macros and we'd get a common type of post from back then.

There's always been a demand for programming by non technical stakeholders that they try and solve without bringing on real programmers. No matter the tool, I think the problem is evergreen.

The way I see this is that it's just another skill differentiator that you can take advantage of if you can get it right.

That is, if it's true that abstraction and architecture are useful for a given product, then people who know how to do those things will succeed in creating that product, and those who don't will fail. I think this is true for essentially all production software, but a lot of software never reaches production.

Transitioning or entirely recreating "vibecoded" proofs of concept to production software is another skill that will be valuable.

Having a good sense for when to do that transition, or when to start building production software from the start, and especially the ability to influence decision makers to agree with you, is another valuable skill.

I do worry about what the careers of entry level people will look like. It isn't obvious to me how they'll naturally develop any of these skills.

> fine and useful

And crippled, incomplete, and deceiving, dangerous.

Why coding agents et al don't make use of the AST through LSP is a question I've been asking myself since the first release of GitHub copilot.

I assume that it's trickier than it seems as it hasn't happened yet.

> Are we sure they know these things as opposed to being able to consistently guess correctly?

What is the practical difference you're imagining between "consistently correct guess" and "knowledge"?

LLMs aren't databases. We have databases. LLMs are probabilistic inference engines. All they do is guess, essentially. The discussion here is about how to get the guess to "check itself" with a firmer idea of "truth". And it turns out that's hard because it requires that the guessing engine know that something needs to be checked in the first place.

Yes. You could ask for factual information like "Tallest building in X place" and first it would answer it did not know. After pressuring it, it would answer with the correct building and height.

But also things where guessing was desirable. For example with a riddle it would tell you it did not know or there wasn't enough information. After pressuring it to answer anyway it would correctly solve the riddle.

The official llama 2 finetune was pretty bad with this stuff.

At that point why not just have LLMs generate bytecode in one shot?

Plenty of training data to go on, I'd imagine.

Would this be addressed by better documentation of code and APIs as well as examples? All this would go into the training materials and then be the body of knowledge.

readability would probably be the sticking point

In most reasonably-sized websites, Tailwind will decrease overall bundle size when compared to other ways of writing CSS. Which is less code, 100 instances of "margin-left: 8px" or 100 instances of "ml-2" (and a single definition for ml-2)? Tailwind will dead-code eliminate all rules you're not using.

In typical production environments tailwind is only around 10kb[1].

[1]: https://v3.tailwindcss.com/docs/optimizing-for-production

You're doing it wrong. Tailwind is endlessly customizable and after compilation is only kilobytes. But yes lets complain because we don't understand the tooling....

I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.

It's certainly annoying, but you can try following up with "can you please remove superfluous comments? In particular, if a comment doesn't add anything to the understanding of the code, it doesn't deserve to be there".

Comments describing the organization and intent, perhaps. Comments just saying what a "require ..." line requires, not so much. (I find it will frequently put notes on the change it is making in comments, contrasting it with the previous state of the code; these aren't helpful at all to anyone doing further work on the result, and I wound up trimming a lot of them off by hand.)

This exact problem is something I’m hoping to fix with a tool that parses the source to AST and then has the LLM write code to modify the AST (which you then run to get your changes) rather than output code directly.

I’ve started in a narrow niche of python/flask webapps and constrained to that stack for now, but if you’re interested I’ve just opened it for signups: https://codeplusequalsai.com

Would love feedback! Especially if you see promising results in not getting huge refactors out of small change requests!

(Edit: I also blogged about how the AST idea works in case you're just that curious: https://codeplusequalsai.com/static/blog/prompting_llms_to_m...)

Asking it explicitly once (not necessarily every new prompt in context) to keep output minimal and strive to do nothing more than it is told works for me.

Can't you just commit the relevant parts? The git index is made for this sort of thing.

Really? I haven't tried Gemini 2.5 yet, but my main complaint with Claude 3.7 is this exact behavior - creating 200+ line diffs when I asked it to fix one function.

I've used it via Google's own AI studio and via my own library/program using the API and finally via Aider. All of them lead to the same outcome, large chunks of changes to a lot of unrelated things ("helpful" refactors that I didn't ask for) and tons of unnecessary comments everywhere (like those comments you ask junior devs to stop making). No amount of prompting seems to address either problems.

No, you can tell it not to write these comments in every prompt and it'll still do it

LLMs are extremely poor at following negative instructions, tell them what to do, not what not to do.

Sample size of one, but I just tried it and it worked for me on 2.5 pro. I just ended my prompt with "Do not include any comments whatsoever."

I usually ask ChatGPT to "comment the shit out of this" for everything it writes. I find it vastly helps future LLM conversations pick up all of the context and why various pieces of code are there.

If it is ingesting data, there should also be a sample of the data in a comment.

Depends on what you mean by "defensive". Anticipating error and non-happy-path cases and handling them is definitely good. Also fault tolerance, i.e. allowing parts of the application to fail without bringing down everything.

But I've heard "defensive code" used for the kind of code where almost every method validates its input parameters, wraps everything in a try-catch, returns nonsensical default values in failure scenarios, etc. This is a complete waste because the caller won't know what to do with the failed validations or thrown errors, and it's just unnecessary bloat that obfuscates the business logic. Validation, error handling and so on should be done in specific parts of the codebase (bonus points if you can encode the successful validation or the presence/absence of errors in the type system).

Does the code consist of many large try except blocks that catch "Exception", which Gemini seems to like doing, (I thought it was a bad practice to catch the generic Exception in Python)

Yep i’ve been using the old and new models in cline. I can’t tell any difference outside of the improvement with diffs, but that’s good enough for me.

They should have shown the benchmarks. Or market it as a coding model, like Qwen & Mistral.

Every frontier model now is a distill of a larger unpublished model. This could be a slightly smaller distill, with potentially the extra tuning you're mentioning.

I appriciate that you are trying to help

But I do not want to have to build a network of bots with non-deterministic outputs to simply stay on top of versions

It's the company. Letting incompetent people who are vocal rise to the top is a part of Google's culture, and the internal performance review process discourages excellence - doing the thousand small improvements that makes a product truly great is invisible to it, so nobody does it.

Software that people truly love is impossible to build in there.

Like what? I use it daily and haven't come across any seriously dysfunctional or incompetent.

another thing i hate is the cmd+enter (Accept) and cmd+del (cancel) being on the same button with tiny ui text and that changes interchangeably depending if its a command or edit or tool call