Developing with GitHub Copilot Agent Mode and MCP

It definitely takes a lot of experience writing code with a LLM. Like a junior engineer it makes tons of (small) mistakes. It takes years of practice to detect the LLM is introducing small bugs that will reveal themselves only after extensive testing or running in prod.

It will be interesting to see how beginning developers will deal with these bugs as they did not write the code and do not have a mental model of the code. Will quality drop? Perhaps some can be compensated by letting the LLM do extensive testing.

Requirements are filters for the set of implementations. The only feedback is the count and the nature of the results. And what you usually do is to either abandoning it or restricting it further. Because the source of the requirements is the business domain which exist outside the technical domain.

Selecting one of the implementation over the other is design, aka making decisions. Sometimes you have to prototype it out to where which parameters is the best. And sometimes a decision can revert an earlier one and you have to investigate the impact radius of that change.

But coding is straightforward translation. The illusion of going faster is that we forego making decisions. Instead we're hoping that the agent makes the correct ones based on some general direction, forgetting than an inch deviation can easily turn into a mile error. The hopeful things would have been an iteration, adding the correct decisions and highlighting the bad ones to avoid. But no one have that kind of patience. And those that use LLMs often finish with a "LGTM" patch.

The normal engineering is to attain a critical mass of decisions and turns that immediately to formal notation which is unambiguous. Then we validate with testing if that abrupt transformation was done properly. But all the decisions were made with proper information.

Chill Winston! Artists in residency are not know for being technical. They are not divas demanding support but individuals who are supposed to have access to resources, space, and support that allows them to develop as artists.

The spaces they are working with often benefit from having talented creatives but this isn't a performance gig we're talking about.

Excellent point! His approach worked in practice, but it would never work in a theoretical situation where proving a point is more important than just solving the problem, so it's obviously worthless.

Dothey do requirement gatherings? Like talking to stakeholder and getting their input of what the feature should, translating business jargon to domain terms?

No.

Do they do the analysis? Removing specs that conflict with each other, validating what's possible in the technical domain and in the business domain?

No.

Do they help with design? Helping coming up with the changes that impact the current software the least, fitting in the current architecture and be maintainable in the feature.

All they do is pattern matching on your prompt and the weights they have. Not a true debate or weighing options based on the organization context.

Do they help with coding?

A lot if you're already experienced with the codebase and the domain. But that's the easiest part of the job.

Do they help with testing? Coming up with tests plan, writing test code, running them, analysing the output of the various tools and producing a cohesive report of the defects?

I don't know as I haven't seen any demo on that front.

Do they help with maintenance? Taking the same software and making changes to keep it churning on new platforms, through dependencies updates and bug fixes?

No demo so far.

The huge gap between the people who claim "It helps me some/most of the time" and the other people who claim "I've tried everything and it's all bad" is really interesting to me.

Is it a problem of knowledge? Is it a problem of hype that makes people over-estimate their productivity? Is it a problem of UX, where it's hard to figure out how to use these tools correctly? Is it a problem of the user's skills, where low-skilled developers see lots of value but high-skilled developers see no value, or even negative value sometimes?

The experiences seem so different, that I'm having a hard time wrapping my mind around it. I find LLMs useful in some particular instances, but not all of them, and I don't see them as the second coming of Jesus. But then I keep seeing people saying they've tried all the tools, and all the approaches, and they understand prompting, yet they cannot get any value whatsoever from the tools.

This is maybe a bit out there, but would anyone (including parent) be up for sending me a screen recording of exactly what you're doing, if you're one of the people that get no value whatsoever from using LLMs? Or maybe even a video call sharing your screen?

I'm not working in the space, have no products or services to sell, only curious is why this vast gap seemingly exists, and my only motive would be to understand if I'm the one who is missing something, or there are more effective ways to help people understand how they can use LLMs and what they can use them for.

My email is on my profile if anyone is up for it. Invitation open for anyone struggling to get any useful responses from LLMs.

Stack overflow didn't create a positive feedback loop where the solution to having to deal with an obscure, badly written, incomprehensible code base is creating an more incomprehensible sloppy code to glue it all together.

Neither did intellisense. If anything, it encouraged structuring your code better so that intellisense would be useful.

Intellisense does little for spaghetti code. And it was my #1 motivation to document the code in a uniform way, too.

The most important impact of tools is that they change the way we think and see the world, and this shapes the world we create with these tools.

When you hold a hammer, everything is a nail, as the saying goes.

And when you hold a gun, you're no longer a mere human; you're a gunman. And the solution space for all sorts of problems starts looking very differently.

The AI debate is not dissimilar to the gun debate.

Yes, both guns and the AI are powerful tools that we have to deal with now that they've been invented. And people wielding these tools have an upper hand over those who don't.

The point that people make in both debates that tends to get ignored by the proponents of these tools is that excessive use of the tools is exacerbating the very problem these tools are ostensibly solving.

Giving guns to all schoolchildren won't solve the problem of high school shootings — it will undeniably make it worse.

And giving the AI to all software developers won't solve the problem of bad, broken code that negatively impacts people who interact with it (as either users or developers).

Finally, a note. Both the gun technology and the AI have been continuously improved since their invention. The progress is undeniable.

Anyone who is thinking about guns in 1850 terms is making a mistake; the Maxim was a game changer. And we're not living in ChatGPT 2.0 times either.

But with all the progress made, the solution space that either tool created hasn't been changing in nature. A problem that wasn't solveable with a flintlock musket or several remains intractable for an AK-74 or an M16.

Improvements in either tech certainly did change the scale at which the tools were applied to resolve all sorts of problems.

And the first half of the 20th century, to this day, provides most of the most brilliant, masterful examples of using guns at scale.

What is also true is that the problems never went away. Nor did better guns made the lives of the common soldier any better.

The work of people like nurse Nightingale did.

And most of that work was that the solution to increasingly devastating battlefield casualties and dropping battlefield effectiveness wasn't giving every soldier a Maxim gun — it was better hygiene and living conditions. Washing hands.

The Maxim gun was a game changer, but it wasn't a solution.

The solution was getting out of the game with stupid prizes (like dying of cholera or typhoid fever). And it was an organizational issue, not a technological one.

* * * * *

To end on a good note, an observation for the AI doomers.

Genocides have predated the guns by millenia, and more people have died by the machete and the bayonet than by any other weapon even in the 20th century. Perhaps the 21st too.

Add disease and famine, and death by gun are a drop in the bucket.

Guns aren't a solution to violence, but they're not, in themselves, a cause of it on a large enough scale.

Mass production of guns made it possible to turn everyone into a soldier (and a target), but the absolute majority of people today have never seen war.

And while guns, by design, are harmful —

— they're also hella fun.