AI Is Like a Crappy Consultant

Honestly I've already had to work with crappier consultants.

Also, there's a lot of value already in a crappy but fast and cheap consultant.

Some of the phenomenon described in this post are felt a lot when using AI.

My own anecdote with a codebase I'm familiar with is indeed, as the article mentions, it's a terrible architect. The problem I was solving ultimately called for a different data structure, but it never had that realization, instead trying to fit the problem shape into an existing, suboptimal way to represent the data.

When I mentioned that this part of the code was memory-sensitive, it indeed wrote good code! ...for the bad data structure. It even included some nice tests that I decided to keep, including memory benchmarks. But the code was ultimately really bad for the problem.

This is related to the sycophancy problem. AI coding assistants bias towards assuming the code they're working with is correct, and that the person using them is also correct. But often neither is ideal! And you can absolutely have a model second-guess your own code and assumptions, but it takes a lot of persistent work because these damn things just want to be "helpful" all the time.

I say all of this as a believer in this paradigm and one who uses these tools every day.

In six months AI has gone from an idiot savant intern to a crappy consultant. I'd call that progress.

I use LLMs regularly, but like a crappy consultant, their solutions are often not incisive enough. The answer I get is frequently 10x longer than I actually want. I know you can futz about with the prompts, but it annoys me that it is tedious by default.

Many companies are perfectly ok with crappy results from a human consultant. Getting those results for fractions of a cent per token? They're on that like flies on a...crappy consultant.

There are tricks one can use to mitigate some of the pitfalls when using either a conversational LLM or a code assistant.

They emerge from the simple assumptions that:

- LLMs fundamentally pattern match bytes. It's stored bytes + user query = generated bytes.

- We have common biases and instinctively use heuristics. And we are aware of some of them. Like confirmation bias or anthropomorphism.

Some tricks:

1. Ask for alternate solutions or let them reword their answers. Make them generate lists of options.

2. When getting an answer that seems right, query for a counterexample or ask it to make the opposite case. This can sometimes help one to remember that we're really just dealing with clever text generation. In other cases it can create tension (I need to research this more deeply or ask an actual expert). Sometimes it will solidify one of the two, answers.

3. Write in a consistent and simple style when using code assistants. They are the most productive and reliable when used as super-auto-complete. They only see the bytes, they can't reason about what you're trying to achieve and they certainly can't read your mind.

4. Let them summarize previous conversations or a code module from time to time. Correct them and add direction whenever they are "off", either with prompts or by adding comments. They simply needed more bytes to look at to produce the right ones at the end.

5. Try to get wrong solutions. Make them fail from time to time, or ask too much of them. This develops a intuition for when these tools work well and when they don't.

6. This is the most important and reflected in the article: Never ask them to make decisions, for the simple fact that they can't do it. They are fundamentally about _generating information_. Prompt them to provide information in the form of text and code so you can make the decisions. Always use them with this mindset.

I think there's something poetic about that fact that you can go on some AI prompt subreddits and have folks there make posts about turning ChatGPT into an "super business consultant" and then go over hear to read about how it's actually pretty bad at that.

But back on point, I found AI works best when given a full set of guardrails around what it should do. The other day I put it to work generating copy for my website. Typically it will go off the deep end if you try to make it generate entire paragraphs but for small pieces of text (id say up to 3 sentences) it does surprisingly well and because it's outputting such small amounts of text you can quickly make edits to remove places where it made a bad word choice or didn't describe something quite right.

But I would say I only got ChatGPT to do this after uploading 3-4 large documents that outline my product in excruciating detail.

As for coding tasks again it works great when given max guardrails. I had several pages that had strings from an object and I wanted those strings to be put back in the code and taken out of the object. This object has ~500 lines in it so it would have taken all day but I ended up doing it in about an hour by having AI do most of the work and just going in after the fact and verifying. This worked really well but I would caution folks that this was a very very specific use case. I've tried vibe coding once for shits and giggles and I got annoyed and stopped after about 10 minutes, IMHO if you're a developer at the "Senior" level, dealing with AI output is more crumbsome than just writing the damn code yourself.

As a once crappy consultant I would say no.

Instant answers, correct or not.

Cheaper per answer by magnitudes.

Solutions provided with extensive documentation.

At the moment, I use Windsurf to explain me how a feature is written and how to do 3rd party integrations. I ask for the approach and I write the code myself. Letting AI write the code has become very unproductive over the period of time.

I'm still learning though

I find that if you talk about architecture it can give excellent advice. It can also refactor in accordance with your existing architecture. If you do not bring up architecture I suppose it could use a bad one though I have not had that issue since I always mention the architecture when I ask it to implement a new feature, which is not "vibe coding". But then why should I vibe code?

Another conclusion is that we could benefit from benchmarks for architectural quality.

AI is like a crappy consultant who doesn't care how many times you reject their code and will get it right if you feed them enough information.

The amount of time I save just by not having to write tests or jsdocs anymore is amazing. Refactoring is amazing.

And that's just the code - I also use AI for video production, 3d model production, generating art and more.

Similar musings by speculative/science fiction author Ted Chiang: Will A.I. Become the New McKinsey? – https://www.newyorker.com/science/annals-of-artificial-intel...