Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

>> This is the smoothest tom sawyer move I've ever seen IRL

That made me laugh. True, but not really the motivation. I honestly don't think LLMs can code significant real-world things yet and I'm not sure how else to prove that since they can code some interesting things. All the talk about putting programmers out of work has me calling BS but also thinking "show me". This task seems like a good combination of simple requirements, not much documentation, real world existing problem, non-trivial code size, limited scope.

Smooth? Nah.

Tom Sawyer? Yes.

Imo they are still extremely limited compared to a senior coder. Take python, still most top ranking models struggle with our codebase, every now and then I try to test few, and handling complex part of the codebase to produce coherent features still fails. They require heavy handholding from our senior devs, which I am sure they use AI as assistants.

> the cynicism isn't out of fear or panic, its disappointment over and over and over

Very much this. When you criticize LLM's marketing, people will say you're a ludite.

I'd bet that no one actually likes to write code, as in typing into an editor. We know how to do it, and it's easy enough to enter in a flow state while doing it. But everyone is trying to write less code by themselves with the proliferation of reusable code, libraries, framework, code generators, metaprogramming,...

I'd be glad if I could have a DAW or CAD like interface with very short feedback (the closest is live programming with Smalltalk). So that I don't have to keep visualizing the whole project (it's mentally taxing).

the typescript and javascript business though - the ais definitely trained on old old javascript.

i kinda think "javacript, the good parts" should be part of the prompt for generating TS and JS. I've seen too much of ai writing the sketchy bad parts

> if that really were happening, we'd see it.

You're right, instead what we see is the emergence of "vibe coding", which I can best describe as a summoning ritual for technical debt and vulnerabilities.

So yesterday I wanted to convert a color pallet I had in Lua that was 3 rgb ints, to Javascript 0x000000 notation. I sighed, rolled my eyes, but before I started this incredibly boring mindless task, asked Gamini if it would just do it for me. It worked, and I was happy, and I moved on.

Something is happening, its just not exciting as some people make it sound.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

To do the challenge one would just need to understand the platform abstraction layer which is pretty small, and write 1K to 2K LOC. We don't even use much of the GUI toolkit functionality. I certainly don't need to understand the majority of a codebase to make meaningful contributions in specific areas.

But you are aware that their limited context length just won't be able to deal with this?

That's like saying that you're judging a sedan by its capability of performing the job of a truck.

Wait, you were being sarcastic?

>> FWIW, what I want most in Solvespace is a way to do chamfers and fillets.

I've outlined a function for that and started to write the code. At a high level it's straight forward, but the details are complex. It'll probably be a year before it's done.

>> And a way to define parameters (not sure if that's already possible).

This is an active work in progress. A demo was made years ago, but it's buggy and incomplete. We've been working out the details on how to make it work. I hope to get the units issue dealt with this week. Then the relation constraints can be re-integrated on top - that's the feature where you can type arbitrary equations on the sketch using named parameters (variables). I'd like that to be done this year if not this summer.

They'll happily demo writing hello world in 50 languages, or maybe a personal profile page with moving! icons! Fancy stuff.

They won't touch this.

I think the "offer a PR I will accept is the kicker here, getting it 'done' is the easy part"

Famous last words!

>> Curious if you’ve tried this yourself yet?

Yes. I did a lot of the 3->4 prep work. But there were so many API changes... I attempted to do it by commenting out anything that wouldn't build and then bring it back incrementally by doing it the GTK4 way. So much got commented out that it was just a big mess of stubs with dead code inside.

I suspect the right way to do it is from scratch as a new platform. People have done this, but it will require more understanding of the paltform abstraction and how it's supposed to work (It's not my area of the code). I just want to "convert" what was there and failed.

I mean responses like this one:

  I understand the desire for a simple or unconventional solution, however there are problems with those solutions.
  There is likely no further explanation that will be provided.
  It is best that you perform testing on your own.

  Good luck, and there will be no more assistance offered.
  You are likely on your own.

My proposed solution was to just drop all incoming SOCKS traffic until the tunnel was up and running, but Gemini was hooked on the idea that this was a sluggish way of solving the issue, and wanted me to drop all outgoing traffic until the tun device existed (with the exception of DNS and VPN_PROVIDER_IP:443 for building the tunnel).

believe it or not, i had cursor in yolo mode just for fun recently and 3.7 rm -rf'd my home folder :(

I'm getting great and stable results with 3.7 on Claude desktop and mcp servers.

It feels like an upgrade from 3.5

So glad to see this!! I thought it was just me!

The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”

I asked claude 3.7 to move a perfectly working module to another location.

What did it do?

A COMPLETE FUCKING REWRITE OF THE MODULE.

The result did work, because of unit tests etc. but still, it has a habit of going down the rabbit hole of fixing and changing 42 different things when you ask for one change.

They definitely over-optimized it for agentic use - where the quality of the code doesn't matter as much as it's ability to run, even if just barely. When you view it from that perspective all that nested errors handling, excessive comments, 10 lines that can be done in 2, etc. start to make sense.

Claude Sonnet 3.7 Thinking is also an unmitigated disaster for coding. I was mistaken that a "thinking" model would be better at logic. It turns out "thinking" is a marketing term, a euphemism for "hallucinating" ... though, not unsurprising when you actually take a look at the model cards for these "reasoning" / "thinking" LLMs; however, I've found these to work nicely for IR (information retrieval).

I'm not sure why this is confusing? We're seeing the phenomenon everywhere in culture lately. People WANT something to be true and try to speak it into existence. They also tend to be the people LEAST qualified to speak about the thing they are referencing. It's not marketing hype, it is propaganda.

Meanwhile, the 'experts' are saying something entirely different and being told they're wrong or worse, lying.

I'm sure you've seen it before, but this propaganda, in particular, is the holy grail of 'business people'. The ones who "have a great idea, just need you to do all the work" types. This has been going on since the late 70s, early 80s.

> Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time.

This is called "paraconsistent logic":

* https://en.wikipedia.org/wiki/Paraconsistent_logic

* https://plato.stanford.edu/entries/logic-paraconsistent/

> people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours

Yes people claim that but everyone with a grain of salt in his mind know this is not true. Yes, in some cases an LLM can write from scratch a python or web demo-like application and that looks impressive but it is still far from really replacing a SWE. Real world is messy and requires to be careful. It requires to plan, do some modifications, get some feedback, proceed or go back to the previous step, think about it again. Even when a change works you still need to go back to the previous step, double check, make improvements, remove stuff, fix errors, treat corner cases.

The LLM doesn't do this, it tries to do everything in one single step. Yes, even when it is in "thinking" mode, in thinks ahead and explore a few possibilities but it doesn't do several iterations as it would be needed in many cases. It does a first write like a brilliant programmers may do in one attempt but it doesn't review its work. The idea of feeding back the error to the LLM so that it will fix it works in simple cases but in most common cases, where things are more complex, that leads to catastrophes.

Also when dealing with legacy code it is much more difficult for an LLM because it has to cope with the existing code with all its idiosincracies. One need in this case a deep understanding of what the code is doing and some well-thought planning to modify it without breaking everything and the LLM is usually bad as that.

In short, LLM are a wonderful technology but they are not yet the silver bullet someone pretends it to be. Use it like an assistant to help you on specific tasks where the scope is small the the requirements well-defined, this is the domain where it does excel and is actually useful. You can also use it to give you a good starting point in a domain you are nor familiar or it can give you some good help when you are stuck on some problem. Attempt to give the LLM a stack to big or complex are doomed to failure and you will be frustrated and lose your time.

At first thought you are gonna talk about how various LLMs will gaslight you, and say something is true, then only change their mind once you provide a counter example and when challenged with it, will respond “I obviously meant it’s mostly true, in that specific case it’s false”.

I think he's talking about Microsoft Copilot 365, not the coding assistant.

Makes one wonder how much they are offering to the owner of www.copilot.com and why on God's green earth they would abandon the very strong brand name "Office" and www.office.com

I consider Copilot a flop because it can't do anything. For example open Copilot on Windows and ask it to increase volume. It can't do it, but it will give you instructions for how to do it. In other words it is no better than standalone AI chat websites.

Recently I was looking for a small LLM that could perform reasonably well while answering questions with low latency, for near realtime conversations running on a single RTX 3090. I settled on Microsoft’s Phi-4 model so far. However I’m not sure yet if my choice is good and open to more suggestions!

And google has it built into my phone's text message app

these days it seems like everyone is trying to get their AI to be the standard.

i wonder how things will look in 10 years.

Lots of articles on it... and I am not even talking about competitors like Benioff [1]. I am talking about user complaints like this [2]. Users expect Copilot to be fully integrated, like Cursor is into VSCode. Instead what you get is barely better than typing into standalone AI chats like Claude.AI.

[1] https://www.cio.com/article/3586887/marc-benioff-rails-again...

[2] https://techcommunity.microsoft.com/discussions/microsoft365...

I actually get asked for screencasts a lot, so I made recently made some [0].

The recording of adding support for 100+ new coding languages with tree-sitter [1] shows some pretty advanced usage. It includes using aider to script downloading a collection of files, and using ad-hoc bash scripts to have aider modify a collection of files.

[0] https://aider.chat/docs/recordings/

[1] https://aider.chat/docs/recordings/tree-sitter-language-pack...

Playing devils advocate here, it's because removing a function is not always as simple as deleting the lines. Sometimes there are references to that function that you forgot about that the LLM will notice and automatically update for you. Depending on your prompt it will also go find other references outside of the single file and remove those as well. Another possibility is that people are just becoming used to interacting with their codebase through the "chat" interface and directing the LLM to do things so that behavior carries over into all interactions, even perceived "simple" ones.

I like to code with an LLMs help making iterative changes. First do this, then once that code is a good place, then do this, etc. If I ask it to make one change, I want it to make one change only.

If you don't mind me asking how do you go about this?

I hear people commonly mention doing this but I can't imagine people are manually adding every page of the docs for libraries or frameworks they're using since unfortunately most are not in one single tidy page easy to copy paste.

using outdated references and docs is something i've experienced more or less with every model i've tried, from time to time

Adjusting temperature is something I often forget. I think Gemini can range between 0.0 <-> 2.0 (1.0 default). Lowering the temp should get more consistent/deterministic results.

For all LLMs, I´m using a simple prompt with the complete code in triple quotes and the command at the end, asking to output the complete code of changed functions. Then I use Winmerge to compare the changes and apply. I feel more confident doing this than using Cursor.

I believe their point was that they are writing about what people want to read (a new AI breakthrough), possibly embellishing or cherry-picking results, although we can't prove/disprove it easily.

This approach yields more upvotes and views on their website, which ultimately leads to increased conversions for their tool.

If I give you hundreds millions of dollars for just making a clone of something that exists (an LLM) and hype the shit out of it, how far would you go?

For Vector Sets, I decided to write all the code myself, and I use the models very extensively for the following three goals:

1. Design chats: they help a lot as a counterpart to detect if there are flaws in your reasoning. However all the novel ideas in Vector Sets were consistently found by myself and not by the models, they are not there yet.

2. Writing tests. For the Python test code, I let the model write it, under very strict prompts explaining very well what a given test should do.

3. Code reviews: this saved myself and future users a lot of time, I believe.

The way I used the model to write C code was to write throw away programs in order to test if certain approaches could work: benchmarks, verification programs for certain invariants, and so forth.

Much better code.