The copilot delusion

I don't mind your rebuttal of the article, but to suggest that this particular article is AI generated is foolish. The style the author presents is vivid, uses powerful imagery and metaphor and finally, at times, is genuinely funny. More qualitatively, the author incorporates a unique identity that persists throughout the entirety of a long form essay.

All of that is still difficult to get an LLM to do. This isn't AI generated. It's just good writing. Whether you buy the premise or not.

Fundamentally, mission critical low level code isn't the kind of software i want to write anyway. I don't find AI tools super useful for most of the same reasons as the author, but i do kind of get tired of the idea that if you're not writing systems in C you're not really programming.

I like writing front end code. I'm probably never going to have a job where i need or would even want to write a low level graphics library from scratch. Fine, I'm not red-eyed 3am hacker brained, but I'm passionate and good at what i do. I don't think a world where every person working in software has the author's mentality is realistic or desirable.

I sometimes imagine, even before LLMs, how much human-written code still matters, like the DNA that has survived evolution. It's got to be a super low percentage.

The trend with Autocomplete Industrialization (AI) is just speeding up the creation of shanty towns of code, as opposed to architecturally robust foundations. The survivability of code is dropping because of the Copilotz. But perhaps this rapidity of creating crude solutions will increase the chances of something truly significant?

p.p.s there are too many times "it's" should be "its" in that blog post to have been AI-generated. That's a different kind of irony IMO, especially since that's just tricky English syntax (the thing AI is supposed to be good at). Maybe the author used AI to come up with the snarky metaphors. I asked ChatGPT for a sarcastic meaning for AI that starts with Autocomplete :)

from TFA:

> Maybe you’ll never write the code that keeps a plane in the sky. Maybe you’ll never push bits that hold a human life in the balance. Fine. Most don’t. But even if you're just slapping together another CRUD app for some bloated enterprise, you still owe your users respect. You owe them dignity.

Yeah, and the article talks about those ways in which AI is useful. Overall, the author doesn’t have a problem with experts using AI to help them. The main argument is that we’re calling AI a copilot, and many newbies may be trusting it or leaning on it too much, when in reality, it’s still a shitty coworker half the time. Real copilots are actually your peers and experts at what they do.

> Now? We’re building a world where that curiosity gets lobotomized at the door. Some poor bastard—born to be great—is going to get told to "review this AI-generated patchset" for eight hours a day, until all that wonder calcifies into apathy. The terminal will become a spreadsheet. The debugger a coffin.

On the other hand, one could argue that AI is just another abstraction. After all, some folks may complain that over-reliance on garbage collectors means that newbies never learn how to properly manage memory. While memory management is useful knowledge for most programmers, it rarely practically comes up for many modern professional tasks. That said, at least knowing about it means you have a deeper level of understanding and mastery of programming. Over time, all those small, rare details add up, and you may become an expert.

I think AI is in a different class because it’s an extremely leaky abstraction.

We use many abstractions every day. A web developer really doesn’t need to know how deeper levels of the stack work — the abstractions are very strong. Sure, you’ll want to know about networking and how browsers work to operate at a very high level, but you can absolutely write very nice, scalable websites and products with more limited knowledge. The key thing is that you know what you’re building on, and you know where to go learn about things if you need to. (Kind of like how a web developer should know the fundamental basics of HTML/CSS/JS before really using a web framework. And that doesn’t take much effort.)

AI is different — you can potentially get away with not knowing the fundamental basics of programming… to a point. You can get away with not knowing where to look for answers and how to learn. After all, AIs would be fucking great at completing basic programming assignments at the college level.

But at some point, the abstraction gets very leaky. Your code will break in unexpected ways. And the core worry for many is that fewer and fewer new developers will be learning the debugging, thinking, and self-learning skills which are honestly CRITICAL to becoming an expert in this field.

You get skills like that by doing things yourself and banging your head against the wall and trying again until it works, and by being exposed to a wide variety of projects and challenges. Honestly, that’s just how learning works — repetition and practice!

But if we’re abstracting away the very act of learning, it is fair to wonder how much that will hurt the long-term skills of many developers.

Of course, I’m not saying AI causes everyone to become clueless. There are still smart, driven people who will pick up core skills along the way. But it seems pretty plausible that the % of people who do that will decrease. You don’t get those skills unless you’re challenged, and with AI, those beginner level “learn how to program” challenges become trivial. Which means people will have to challenge themselves.

And ultimately, the abstraction is just leaky. AI might look like it solves your problems for you to a novice, but once you see through the mirage, you realize that you cannot abstract away your core programming & debugging skills. You actually have to rely on those skills to fix the issues AI creates for you — so you better be learning them along the way!!

Btw, I say this as someone who does use AI coding assistants. I don’t think it’s all bad or all good. But we can’t just wave away the downsides just because it’s useful

It’s funny because I made a few funny clips (to my taste) on Google Whisk and figured, hey why not, let’s make a TikTok. Did you know that all of TikTok is full of millions of ai generated stuff or other people just copying each others stuff? I really thought there was something to this “original creation” stuff.

We are all so simply reproducible. No one’s making anything special, anywhere, for the most part. If we all uploaded a TikTok video of daily coding, it would be the same fucking app over and over, just like the rest of TikTok.

Elon may have be right all along, there’s literally nothing left to do but goto Mars. Some of us were telling many of you that the LLMs don’t hallucinate as much as you think just two years ago, and I think the late to the party crowd need to hear us again - we humans are not really necessary anymore.

!RemindMe in 2 years

Same, The author writes like Dave Barry. I burst out laughing more than once. He was able to articulate with a lot of humor exactly what I think of co-pilot.

Exactly. I think the key point from the article is: "When you outsource the thinking, you outsource the learning."

I feel privileged to be able to say that as a software engineer who's been doing it the hard way for 20+ years, I relish the hard problems. The CRUD app updates are unbearable without the random in-between challenges that bend my mind. The rare recursive algorithm, the application of some esoteric knowledge I actually learned in college, actually having to do big-o estimates. These are the gems of my career that keep me sane. I hope the next flock of AI-driven SWEs appreciates these things even more given the AI can spout off answers which sometimes are right and sometimes are horribly wrong. Challenges like these will always have to have someone who actually knows what to do when the AIs start hallucinating or the context of the situation is beyond the context window.

> Management has an AI shaped hammer and they're hitting everything to see if it's a nail.

I really think we need to figure out how to cut back on management so we can get back to the business of actually doing work

Yeah, I mean, this is just the current phase of the hype cycle. It'll settle down. Some of the tools and techniques will have staying power, most won't. If you can figure out which is which and influence others, you'll be in good shape.

> I feel this in my bones. Every day I'm getting challenged by leadership that we're not using AI enough, told that I should halve my estimates because "we'll use AI", and being told that there's a new AI tool that I have to adopt because someone is tracking KPIs related to adoption and if our team doesn't adopt enough AI tools we're going to be fired to give more headcount to those that do.

This--all of this--seems exactly antithetical to computing/development/design/"engineering"/architecture/whatever-the-hell people call this profession as I understood it.

Typically, I labored under the delusion that competent technical decision makers would integrate tooling or choose to use a language, "service", platform, whatever, if they saw benefits and if they could make a "case" for why something was the correct approach, i.e how it met some product's needs, addressed some shortcomings, made things more efficient.

Like "here's my design doc, I chose $THING for caching for $REASON and $DATASTORE as it offers blah blah"

"Please provide feedback and questions"

This is totally alien to that approach.

Ideally, "hey we're going to use CoPilot/other LLM thingy, let us know if it aids your workflow, give us some report in a month and we'll go from there to determine if we want to keep paying for it"

> AI is always being touted as the tool to replace the other guy's job. But in reality it only appears to do a good job because you don't understand the other guy's job.

This is a well considered point that not enough of us admit. Yes many jobs are rote or repetitive, but many more jobs, of all flavors, done well have subtleties that will be lost when things are automated. And no I do not think that some "80% done by AI is good enough" because errors propagate through a system (even if that system is a company or society), AND the people evaluating that "good enough" are not necessarily going to be those experienced in that same domain.

But, management is the one to go soon. The other shoe is going to drop dear brother, this I promise you. Stay strong.

Well when you have a hammer big enough everything is indeed a nail.

Have you considered that instead of resisting you should do more to figure out why you're not getting the results that a lot of us are talking about? If nothing has changed for you in the past 2 years in your productivity the problem is most likely you. Don't you think it's your responsibility as an engineer to figure out what you're doing wrong when there are a lot of people telling you that it's a life changing tool? Or did you just assume that everybody was lying and you were doing everything correctly?

Sorry to say it. It's an unpopular opinion but I think it's pretty much a harsh truth.

They seem to be a game dev judging by their other posts. I imagine there’s a lot less content online about that for LLMs to scrape than yet another CRUD app.

That’s probably asking for too much but I agree.

Here are some terms/aspects of LLMs that people _regularly_ use, yet 10 people have 10 definitions of what it means (to them)

- Vibe Coding

- Boilerplate

- Copilot

- Cursor/Aider/Claude Code/Codex/OpenHands/etc

- LLM Autocomplete and/or inline code suggestion

- LLM Agent

I’m happy to explain or expand on any of those if it’s not clear what I mean.