Gemini 2.5 Pro Preview
376 comments
·May 6, 2025segphault
jstummbillig
> no amount of prompting will get current models to approach abstraction and architecture the way a person does
I find this sentiment increasingly worrisome. It's entirely clear that every last human will be beaten on code design in the upcoming years (I am not going to argue if it's 1 or 5 years away, who cares?)
I wished people would just stop holding on to what amounts to nothing, and think and talk more about what can be done in a new world. We need good ideas and I think this could be a place to advance them.
epolanski
> no amount of prompting will get current models to approach abstraction and architecture the way a person does
Which person it is? Because 90% of the people in our trade are bad, like, real bad. Most of us don't get that we care that bit more to end up meeting less of them, but it happens.
DanHulton
> It's entirely clear that every last human will be beaten on code design in the upcoming years
Citation needed. In fact, I think this pretty clearly hits the "extraordinary claims require extraordinary evidence" bar.
pinoy420
[dead]
coffeemug
AlphaGo.
Workaccount2
Software will change to accommodate LLMs, if for no other reason than we are on the cusp of everyone being a junior level programmer. What does software written for LLMs to middleman look like?
I think there is a total seismic change in software that is about to go down, similar to something like going from gas lamps to electric. Software doesn't need to be the way it is now anymore, since we have just about solved human language to computer interface translation. I don't want to fuss with formatting a word document anymore, I would rather just tell and LLM and let it modify the program memory to implement what I want.
jjice
I'm confused by your comment. It seems like you didn't really provide a retort to the parent's comment about bad architecture and abstraction from LLMs.
FWIW, I think you're probably right that we need to adapt, but there was no explanation as to _why_ you believe that that's the case.
TuringNYC
I think they are pointing out that the advantage humans have has been chipped away little by little and computers winning at coding is inevitable on some timeline. They are also suggesting that perhaps the GP is being defensive.
mattgreenrocks
I'm always impressed by the ability of the comment section to come up with more reasons why decent design and architecture of source code just can't happen:
* "it's too hard!"
* "my coworkers will just ruin it"
* "startups need to pursue PMF, not architecture"
* "good design doesn't get you promoted"
And now we have "AI will do it better soon."
None of those are entirely wrong. They're not entirely correct, either.
davidsainez
I use LLMs for coding every day. There have been significant improvements over the years but mostly across a single dimension: mapping human language to code. This capability is robust, but you still have to know how to manage context to keep them focused. I still have to direct them to consider e.g. performance or architecture considerations.
I'm not convinced that they can reason effectively (see the ARC-AGI-2 benchmarks). Doesn't mean that they are not useful, but they have their limitations. I suspect we still need to discover tech distinct from LLMs to get closer to what a human brain does.
acedTrex
> It's entirely clear that every last human will be beaten on code design in the upcoming years
In what world is this statement remotely true.
saurik
I mean, didn't you just admit you are wrong? If we are talking 1-5 years out, that's not "current models".
jstummbillig
Imagine sitting in a car, that is fast approaching a cliff, with no brakes, while the driver talks about how they have not been in any serious car accident so far.
Technically correct. And yet, you would probably be at least be a little worried about that cliff and rather talk about that.
Jordan-117
I recently needed to recommend some IAM permissions for an assistant on a hobby project; not complete access but just enough to do what was required. Was rusty with the console and didn't have direct access to it at the time, but figured it was a solid use case for LLMs since AWS is so ubiquitous and well-documented. I actually queried 4o, 3.7 Sonnet, and Gemini 2.5 for recommendations, stripped the list of duplicates, then passed the result to Gemini to vet and format as JSON. The result was perfectly formatted... and still contained a bunch of non-existent permissions. My first time being burned by a hallucination IRL, but just goes to show that even the latest models working in concert on a very well-defined problem space can screw up.
darepublic
Listen I don't blame any mortal being for not grokking the AWS and Google docs. They are a twisting labyrinth of pointers to pointers some of them deprecated though recommended by Google itself.
dotancohen
AWS docs have (had) an embedded AI model that would do this perfectly. I suppose it had better training data, and the actual spec as a RAG.
djhn
Both AWS and Azure docs’ built in models have been absolutely useless.
siscia
This problem have been solved by LSP (language server protocol), all we need is a small server behind MCP that can communicate LSP information back to the LLM and get the LLM to use by adding to the prompt something like: "check your API usage with the LSP"
The unfortunate state of open source funding makes buildings such simple tool a loosing adventure unfortunately.
satvikpendem
This already happens in agent modes in IDEs like Cursor or VSCode with Copilot, it can check for errors with the LSP.
doug_durham
If they never get good at abstraction or architecture they will still provide a tremendous amount of value. I have them do the parts of my job that I don't like. I like doing abstraction and architecture.
mynameisvlad
Sure, but that's not the problem people have with them nor the general criticism. It's that people without the knowledge to do abstraction and architecture don't realize the importance of these things and pretend that "vibe coding" is a reasonable alternative to a well-thought-out project.
Karrot_Kream
We can rewind the clock 10 years and I can substitute "vibe coding" for VBA/Excel macros and we'd get a common type of post from back then.
There's always been a demand for programming by non technical stakeholders that they try and solve without bringing on real programmers. No matter the tool, I think the problem is evergreen.
sanderjd
The way I see this is that it's just another skill differentiator that you can take advantage of if you can get it right.
That is, if it's true that abstraction and architecture are useful for a given product, then people who know how to do those things will succeed in creating that product, and those who don't will fail. I think this is true for essentially all production software, but a lot of software never reaches production.
Transitioning or entirely recreating "vibecoded" proofs of concept to production software is another skill that will be valuable.
Having a good sense for when to do that transition, or when to start building production software from the start, and especially the ability to influence decision makers to agree with you, is another valuable skill.
I do worry about what the careers of entry level people will look like. It isn't obvious to me how they'll naturally develop any of these skills.
yousif_123123
The opposite problem is also true. I was using it to edit code I had that was calling the new openai image API, which is slightly different from the dalle API. But Gemini was consistently "fixing" the OpenAI call even when I explained clearly not to do that since I'm using a new API design etc. Claude wasn't having that issue.
The models are very impressive. But issues like these still make me feel they are still more pattern matching (although there's also some magic, don't get me wrong) but not fully reasoning over everything correctly like you'd expect of a typical human reasoner.
toomuchtodo
It seems like the fix is straightforward (check the output against a machine readable spec before providing it to the user), but perhaps I am a rube. This is no different than me clicking through a search result to the underlying page to verify the veracity of the search result surfaced.
disgruntledphd2
Why coding agents et al don't make use of the AST through LSP is a question I've been asking myself since the first release of GitHub copilot.
I assume that it's trickier than it seems as it hasn't happened yet.
disgruntledphd2
They are definitely pattern matching. Like, that's how we train them, and no matter how many layers of post training you add, you won't get too far from next token prediction.
And that's fine and useful.
mdp2021
> fine and useful
And crippled, incomplete, and deceiving, dangerous.
codebolt
I've found they do a decent job searching for bugs now as well. Just yesterday I had a bug report on a component/page I wasn't familiar with in our Angular app. I simply described the issue as well as I could to Claude and asked politely for help figuring out the cause. It found the exact issue correctly on the first try and came up with a few different suggestions for how to fix it. The solutions weren't quite what I needed but it still saved me a bunch of time just figuring out the error.
M4v3R
That’s my experience as well. Many bugs involve typos, syntax issues or other small errors that LLMs are very good at catching.
onlyrealcuzzo
2.5 pro seems like a huge improvement.
One area I've still noticed weakness is if you want to use a pretty popular library from one language in another language, it has a tendency to think the function signatures in the popular language match the other.
Naively, this seems like a hard problem to solve.
I.e. ask it how to use torchlib in Ruby instead of Python.
jug
I’ve seen benchs on hallucinations and OpenAI has typically performed worse than Google and Anthropic models. Sometimes significantly so. But it doesn’t seem like they have cared much. I’ve suspected that LLM performance is correlated to risking hallucinations? That is, if they’re bolder, this can be beneficial? Which helps in other performance benchmarks. But of course at the risk of hallucinating more…
mountainriver
The hallucinations are a result of RLVR. We reward the model for an answer and then force it to reason about how to get there when the base model may not have that information.
mdp2021
> The hallucinations are a result of RLVR
Well let us reward them for producing output that is consistent with database accessed selected documentation then, and massacre them for output they cannot justify - like we do with humans.
paulirish
> Gemini 2.5 Pro now ranks #1 on the WebDev Arena leaderboard
It'd make sense to rename WebDev Arena to React/Tailwind Arena. Its system prompt requires [1] those technologies and the entire tool breaks when requesting vanilla JS or other frameworks. The second-order implications of models competing on this narrow definition of webdev are rather troublesome.
[1] https://blog.lmarena.ai/blog/2025/webdev-arena/#:~:text=PROM...
aero142
If llms are able to write better code with more declarative and local programming components and tailwind, then I could imagine a future where a new programming language is created to maximize llm success.
martinsnow
Bwoah it's almost as if react and tailwind is the bees knees ind frontend atm
byearthithatius
Sadly. Tailwind is so oof in my opinion. Lets import megabytes just so we don't have to write 5 whole CSS classes. I mean just copy paste the code.
Don't get me stared on how ugly the HTML becomes when most tags have 20 f*cking classes which could have been two.
xd1936
[flagged]
shortcord
Not a fan of the dominance of shadcn and Tailwind when it comes to generating greenfield code.
postalrat
I've found them to be pretty good with vanilla html and css.
ranyume
I don't know if I'm doing something wrong, but every time I ask gemini 2.5 for code it outputs SO MANY comments. An exaggerated amount of comments. Sections comments, step comments, block comments, inline comments, all the gang.
lukeschlather
I usually remove the comments by hand. It's actually pretty helpful, it ensures I've reviewed every piece of code carefully, especially since most of the comments are literally just restating the next line, and "does this comment add any information?" is a really helpful question to make sure I understand the code.
tasuki
Same! It eases my code review. In the rare occasions I don't want to do that, I ask the LLM to provide the code without comments.
Benjammer
I've found that heavily commented code can be better for the LLM to read later, so it pulls in explanatory comments into context at the same time as reading code, similar to pulling in @docs, so maybe it's doing that on purpose?
koakuma-chan
No, it's just bad. I've been writing a lot of Python code past two days with Gemini 2.5 Pro Preview, and all of its code was like:
```python
def whatever():
--- SECTION ONE OF THE CODE ---
...
--- SECTION TWO OF THE CODE ---
try:
[some "dangerous" code]
except Exception as e:
logging.error(f"Failed to save files to {output_path}: {e}")
# Decide whether to raise the error or just warn
# raise IOError(f"Failed to save files to {output_path}: {e}")(it adds commented out code like that all the time, "just in case")
It's terrible.
I'm back to Claude Code.
NeutralForest
I'm seeing it trying to catch blind exceptions in Python all the time. I see it in my colleagues code all the time, it's driving me nuts.
brandall10
It's certainly annoying, but you can try following up with "can you please remove superfluous comments? In particular, if a comment doesn't add anything to the understanding of the code, it doesn't deserve to be there".
null
breppp
I always thought these were there to ground the LLM on the task and produce better code, an artifact of the fact that this will autocomplete better based on past tokens. Similarly always thought this is why ChatGPT always starts every reply with repeating exactly what you asked again
rst
Comments describing the organization and intent, perhaps. Comments just saying what a "require ..." line requires, not so much. (I find it will frequently put notes on the change it is making in comments, contrasting it with the previous state of the code; these aren't helpful at all to anyone doing further work on the result, and I wound up trimming a lot of them off by hand.)
puika
I have the same issue plus unnecessary refactorings (that break functionality). it doesn't matter if I write a whole paragraph in the chat or the prompt explaining I don't want it to change anything else apart from what is required to fulfill my very specific request. It will just go rogue and massacre the entirety of the file.
mgw
This has also been my biggest gripe with Gemini 2.5 Pro. While it is fantastic at one-shotting major new features, when wanting to make smaller iterative changes, it always does big refactors at the same time. I haven't found a way to change that behavior through changes in my prompts.
Claude 3.7 Sonnet is much more restrained and does smaller changes.
cryptoz
This exact problem is something I’m hoping to fix with a tool that parses the source to AST and then has the LLM write code to modify the AST (which you then run to get your changes) rather than output code directly.
I’ve started in a narrow niche of python/flask webapps and constrained to that stack for now, but if you’re interested I’ve just opened it for signups: https://codeplusequalsai.com
Would love feedback! Especially if you see promising results in not getting huge refactors out of small change requests!
(Edit: I also blogged about how the AST idea works in case you're just that curious: https://codeplusequalsai.com/static/blog/prompting_llms_to_m...)
nolist_policy
Can't you just commit the relevant parts? The git index is made for this sort of thing.
fwip
Really? I haven't tried Gemini 2.5 yet, but my main complaint with Claude 3.7 is this exact behavior - creating 200+ line diffs when I asked it to fix one function.
fkyoureadthedoc
Where/how do you use it? I've only tried this model through GitHub Copilot in VS Code and I haven't experienced much changing of random things.
diggan
I've used it via Google's own AI studio and via my own library/program using the API and finally via Aider. All of them lead to the same outcome, large chunks of changes to a lot of unrelated things ("helpful" refactors that I didn't ask for) and tons of unnecessary comments everywhere (like those comments you ask junior devs to stop making). No amount of prompting seems to address either problems.
dherikb
I have the exactly same issue using it with Aider.
bugglebeetle
This is generally controllable with prompting. I usually include something like, “be excessively cautious and conservative in refactoring, only implementing the desired changes” to avoid.
Workaccount2
I have a strong sense that the comments are for the model more than the user. It's effectively more thinking in context.
Maxatar
Tell it not to write so many comments then. You have a great deal of flexibility in dictating the coding style and can even include that style in your system prompt or upload a coding style document and have Gemini use it.
Trasmatta
Every time I ask an LLM to not write comments, it still litters it with comments. Is Gemini better about that?
nearbuy
Sample size of one, but I just tried it and it worked for me on 2.5 pro. I just ended my prompt with "Do not include any comments whatsoever."
grw_
No, you can tell it not to write these comments in every prompt and it'll still do it
sitkack
LLMs are extremely poor at following negative instructions, tell them what to do, not what not to do.
dheera
I usually ask ChatGPT to "comment the shit out of this" for everything it writes. I find it vastly helps future LLM conversations pick up all of the context and why various pieces of code are there.
If it is ingesting data, there should also be a sample of the data in a comment.
HenriNext
Same experience. Especially the "step" comments about the performed changes are super annoying. Here is my prompt-rule to prevent them:
"5. You must never output any comments about the progress or type of changes of your refactoring or generation. Example: you must NOT add comments like: 'Added dependency' or 'Changed to new style' or worst of all 'Keeping existing implementation'."
Semaphor
2.5 was the most impressive model I use, but I agree about the comments. And when refactoring some code it wrote before, it just adds more comments, it becomes like archaeological history (disclaimer: I don’t use it for work, but to see what it can do, so I try to intervene as little as possible, and get it to refactor what it thinks it should)
Scene_Cast2
It also does super defensive coding. Not that it's a bad thing in general, but I write a lot of prototype code.
prpl
Production quality code is defensive. Probably trained on a lot of google code.
montebicyclelo
Does the code consist of many large try except blocks that catch "Exception", which Gemini seems to like doing, (I thought it was a bad practice to catch the generic Exception in Python)
Tainnor
Depends on what you mean by "defensive". Anticipating error and non-happy-path cases and handling them is definitely good. Also fault tolerance, i.e. allowing parts of the application to fail without bringing down everything.
But I've heard "defensive code" used for the kind of code where almost every method validates its input parameters, wraps everything in a try-catch, returns nonsensical default values in failure scenarios, etc. This is a complete waste because the caller won't know what to do with the failed validations or thrown errors, and it's just unnecessary bloat that obfuscates the business logic. Validation, error handling and so on should be done in specific parts of the codebase (bonus points if you can encode the successful validation or the presence/absence of errors in the type system).
andy12_
Interestingly, when compering benchmarks of Experimental 03-25 [1] and Experimental 05-06 [2] it seems the new version scores slightly lower in everything except on LiveCodeBench.
[1] https://storage.googleapis.com/model-cards/documents/gemini-... [2] https://deepmind.google/technologies/gemini/
merksittich
According to the article, "[t]he previous iteration (03-25) now points to the most recent version (05-06)." I assume this applies to both the free tier gemini-2.5-pro-exp-03-25 in the API (which will be used for training) and the paid tier gemini-2.5-pro-preview-03-25.
Fair enough, one could say, as these were all labeled as preview or experimental. Still, considering that the new model is slightly worse across the board in benchmarks (except for LiveCodeBench), it would have been nice to have the option to stick with the older version. Not everyone is using these models for coding.
zurfer
Just switching a pinned version (even alpha, beta, experimental, preview) to another model doesn't feel right.
I get it, chips are sparse and they want their capacity back, but it breaks trust with developers to just downgrade your model.
Call it gemini-latest and I understand that things will change. Call it *-03-25 and I want the same model that I got on 25th March.
arnaudsm
This should be the top comment. Cherry-picking is hurting this industry.
I bet they kept training on coding tasks, made everything worse on the way, and tried to hide it under the rug because of the sunk costs.
nopinsight
Livebench.ai actually suggests the new version is better on most things.
jjani
Sounds like they were losing so much money on 2.5-Pro they came up with a forced update that made it cheaper to run. They can't come out with "we've made it worse across the board", nor do they want to be the first to actually raise prices, so instead they made a bit of a distill that's slightly better at coding so they can still spin it positively.
sauwan
I'd be surprised if this was a new base model. It sounds like they just did some post-training RL tuning to make this version specifically stronger for coding, at the expense of other priorities.
jjani
Every frontier model now is a distill of a larger unpublished model. This could be a slightly smaller distill, with potentially the extra tuning you're mentioning.
Workaccount2
Google doesn't pay the nvidia tax. Their TPUs are designed for Gemini and Gemini designed for their TPUs. Google is no doubt paying far less per token than every other AI house.
null
laborcontract
My guess is that they've done a lot of tuning to improve diff based code editing. Gemini 2.5 is fantastic at agentic work, but it still is pretty rough around the edges in terms of generating perfectly matching diffs to edit code. It's probably one of the very few issues with the model. Luckily, aider tracks this.
They measure the old gemini 2.5 generating proper diffs 92% of the time. I bet this goes up to ~95-98% https://aider.chat/docs/leaderboards/
Question for the google peeps who monitor these threads: Is gemini-2.5-pro-exp (free tier) updated as well, or will it go away?
Also, in the blog post, it says:
> The previous iteration (03-25) now points to the most recent version (05-06), so no action is required to use the improved model, and it continues to be available at the same price.
update: I just tried updating the date in the exp model (gemini-2.5-pro-exp-05-06) and that doesnt work.
laborcontract
Update 2: I've been using this model in both aider and cline and I've haven't gotten a diff matching error yet, even with some pretty difficult substitutions across different places in multiple files. The overall feel of this model is nice.
I don't have a formal benchmark but there's a notable improvement in code generation due to this alone.
I've had gemini chug away on plans that have taken ~1 hour to implement. (~80mln tokens spent) A good portion of that energy was spent fixing mistakes made by cline/aider/roo due to search/replace mistakes. If this model gets anywhere close to 100% on diffs then this is a BFD. I estimate this will translate to a 50-75% productivity boost on long context coding tasks. I hope the initial results i'm seeing hold up!
I'm surprised by the reaction in the rest of the thread. A lot unproductive complaining, a lot of off topic stuff, nothing talking about the model itself.
Any thoughts from anyone else using the updated model?
null
null
okdood64
What do you mean by agentic work in this context?
laborcontract
Knowing when to call functions, generating the proper function calling text structure, properly executing functions in sequence, knowing when it's completed its objective, and doing that over an extended context window.
mohsen1
I use Gemini for almost everything. But their model card[1] only compares to o3-mini! In known benchmarks o3 is still ahead:
+------------------------------+---------+--------------+
| Benchmark | o3 | Gemini 2.5 |
| | | Pro |
+------------------------------+---------+--------------+
| ARC-AGI (High Compute) | 87.5% | — |
| GPQA Diamond (Science) | 87.7% | 84.0% |
| AIME 2024 (Math) | 96.7% | 92.0% |
| SWE-bench Verified (Coding) | 71.7% | 63.8% |
| Codeforces Elo Rating | 2727 | — |
| MMMU (Visual Reasoning) | 82.9% | 81.7% |
| MathVista (Visual Math) | 86.8% | — |
| Humanity’s Last Exam | 26.6% | 18.8% |
+------------------------------+---------+--------------+
jsnell
The text in the model card says the results are from March (including the Gemini 2.5 Pro results), and o3 wasn't released yet.
Is this maybe not the updated card, even though the blog post claims there is one? Sure, the timestamp is in late April, but I seem to remember that the first model card for 2.5 Pro was only released in the last couple of weeks.
null
wewewedxfgdf
Gemini does not accept upload of TSX files, it says "File type unsupported"
You must rename your files to .tsx.txt THEN IT ACCEPTS THEM and works perfectly fine writing TSX code.
This is absolutely bananas. How can such a powerful coding engine have this behavior?
krat0sprakhar
Where are you testing this? I'm able to upload tsx files on aistudio
alana314
The google sheets UI asked me to try Gemini to create a formula, so I tried it, starting with "Create a formula...", and its answer was "Sorry, I can't help with creating formulas yet, but I'm still learning."
cadamsdotcom
Google/Alphabet is a giant hulking machine that’s been frankly running at idle. All that resume driven development and performance review promo cycles and retention of top talent mainly to work on ad tech means it’s packed to the rafters with latent capability. Holding on to so much talent in the face of basically having nothing to do is a testament to the company’s leadership - even if said leadership didn’t manage to make Google push humanity forward over the last decade or so.
Now there’s a big nugget to chew (LLMs) you’re seeing that latent capability come to life. This awakening feels more bottom-up driven than top down. Google’s a war machine chugging along nicely in peacetime, but now its war again!
Hats off to the engineers working on the tech. Excited to try it out!
herpdyderp
I agree it's very good but the UI is still usually an unusable, scroll-jacking disaster. I've found it's best to let a chat sit for around a few minutes after it has finished printing the AI's output. Finding the `ms-code-block` element in dev tools and logging `$0.textContext` is reliable too.
uh_uh
Noticed this too. There's something funny about billion dollar models being handicapped by stuck buttons.
energy123
The Gemini app has a number of severe bugs that impacts everyone who uses it, and those bugs have persisted for over 6 months.
There's something seriously dysfunctional and incompetent about the team that built that web app. What a way to waste the best LLM in the world.
kubb
It's the company. Letting incompetent people who are vocal rise to the top is a part of Google's culture, and the internal performance review process discourages excellence - doing the thousand small improvements that makes a product truly great is invisible to it, so nobody does it.
Software that people truly love is impossible to build in there.
OsrsNeedsf2P
Loading the UI on mobile while on low bandwidth is also a non-starter. It simply doesn't work.
arnaudsm
Be careful, this model is worse than 03-25 in 10 of the 12 benchmarks (!)
I bet they kept training on coding, made everything worse on the way, and tried to hide it under the rug because of the sunk costs.
null
jstummbillig
It seems that trying to build llms is the definition of accepting sunk cost.
ionwake
Is it possible to sue this with Cursor? If so what is the name of the model? gemini-2.5-pro-preview ?
edit> Its gemini-2.5-pro-preview-05-06
edit>Cursor syas it doesnt have "good support" et, but im not sure if this is a defualt message when it doesnt recognise a model? is this a big deal? should I wait until its officially supported by cursor?
Just trying to save time here for everyone - anyone know the answer?
androng
At the bottom of the article it says no action is required and the Gemini-2.5-pro-preview-03-25 now points to the new model
bn-l
The one with exp in the name is free (you may have to add it yourself) but they train on you. And after a certain limit it becomes paid).
tough
Cursor UI sucks, it tells me to use -auto mode- to be faster, but gemini 2.5 is way faster than any of the other free models, so just selecting that one is faster even if the UI says otherwise
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried.
There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person does. But I'm finding that these Gemini models are finally able to replace searches and stackoverflow for a lot of my day-to-day programming.