AI Blindspots – Blindspots in LLMs I've noticed while AI coding

I think it's not very helpful to just declare that such a model doesn't exist: there's a decent amount of evidence that LLMs do in fact form models of the world internally, and use them during inference. However, while these models are very large and complex, they aren't necessarily accurate and LLMs struggle with actually manipulating them at inference time, forming new models or adjusting existing ones is generally something they are quite bad at at those stages (which generally results in the 'high knowledge' which impresses people and is often confused with intelligence, while they're still fundamentally quite dumb despite having a huge depth of knowledge: I don't think it's something you can categorically say 'zero intelligence' - even relatively simpler and less capable systems can be said to have some intelligence, it's just in many aspects LLM intelligence is still worse than a good fraction of mammals)

> they don't have a model of the world

Must it have one? The words "artificial intelligence" are a poor description of a thing when we've not rigorously defined it. It's certainly artificial, there's no question about that, but is it intelligent? It can do all sorts of things that we consider a feature of intelligence and pass all sorts of tests, but it also falls down flat on its face when prompted with a just-so brainteaser. It's certainly useful, for some people. If, by having inhaled all of the Internet and written books that have been scanned as its training data, it's able to generate essays on anything and everything, at the drop of a hat, why does it matter if we can find a brainteaser it hasn't seen yet? It's like it has a ginormous box of Legos, and it can build whatever your ask for with these Lego blocks, but pointing out it's unable create its own Lego blocks from scratch has somehow become critically important to point out, as if that makes this all total dead end and it's all a waste of money omg people wake up oh if only they'd listen to me. Why don't people listen to me?

Crows are believed to have a theory of mind, and they can count up to 30. I haven't tried it with Claude, but I'm pretty sure it can count at least that high. LLMs are artificial, they're alien, of course they're going to look different. In the analogy where they're simply a next word guesser, one imagines standing at a fridge with a bag of magnetic words, and just pulling a random one from the bag to make ChatGPT. But when you put your hand inside a bag inside a bag inside a bag, twenty times (to represent the dozens of layers in an LLM model), and there are a few hundred million pieces in each bag (for parameters per layer), one imagines that there's a difference; some sort of leap, similar to when life evolved from being a single celled bacterium to a multi-cellular organism.

Or maybe we're all just rubes, and some PhD's have conned the world into giving them a bunch of money, because they figured out how to represent essays as a math problem, then wrote some code to solve them, like they did with chess.

LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.

Along these lines one model that might help is to consider LLMs 'wikipedia of all possible correct articles'. Start with Wikipedia and assume (already a tricky proposition!) that it's perfectly correct. Then, begin resynthesizing articles based on what's already there. Do your made-up articles have correctness?

I'm going to guess that sometimes they will: driven onto areas where there's no existing article, some of the time you'll get made-up stuff that follows the existing shapes of correct articles and produces articles that upon investigation will turn out to be correct. You'll also reproduce existing articles: in the world of creating art, you're just ripping them off, but in the world of Wikipedia articles you're repeating a correct thing (or the closest facsimile that process can produce)

When you get into articles on exceptions or new discoveries, there's trouble. It can't resynthesize the new thing: the 'tokens' aren't there to represent it. The reality is the hallucination, but an unreachable one.

So the LLMs can be great at fooling people by presenting 'new' responses that fall into recognized patterns because they're a machine for doing that, and Turing's Test is good at tracking how that goes, but people have a tendency to think if they're reading preprogrammed words based on a simple algorithm (think 'Eliza') they're confronting an intelligence, a person.

They're going to be historically bad at spotting Holmes-like clues that their expected 'pattern' is awry. The circumstantial evidence of a trout in the milk might lead a human to conclude the milk is adulterated with water as a nefarious scheme, but to an LLM that's a hallucination on par with a stone in the milk: it's going to have a hell of a time 'jumping' to a consistent but very uncommon interpretation, and if it does get there it'll constantly be gaslighting itself and offering other explanations than the truth.

> overconfidence is the problem.

The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion.

The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics.

In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text.

Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers."

> Hallucinating

P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.

Hallucinating is a confidence problem, no?

Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident.

On a much more nuanced problem, "0.30000000000000004" may get more and more confidence.

This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )

Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.

Inventiveness/creativity/imagination are deliberate things. LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. More "willful hope in the face of forgetting" than "creativity." Many LLM hallucinations - especially of the coding sort - are ones that would be obviously-wrong based on the training material, but the hundreds of languages/libraries/frameworks the thing was trained on start to blur together and there is not precise 100%-memorization recall but instead a "probably something like this" guess.

It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details.

If and only if the LLM is able to bring the novel, unexpected connection into itself and see whether it forms other consistent networks that lead to newly common associations and paths.

A lot of us have had that experience. We use that ability to distinguish between 'genius thinkers' and 'kid overdosing on DMT'. It's not the ability to turn up the weird connections and go 'ooooh sparkly', it's whether you can build new associations that prove to be structurally sound.

If that turns out to be something self-modifying large models (not necessarily 'language' models!) can do, that'll be important indeed. I don't see fiddling with the 'temperature' as the same thing, that's more like the DMT analogy.

You can make the static model take a trip all you like, but if nothing changes nothing changes.

> What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ...

No.

What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding.

0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator

the word you're looking for is "confabulation"

But it's different kinds of hallucinations.

AI doesn't have a base understanding of how physics work. So they think it's acceptible if in a video some element on the background in a next frame might appear in front of another element that is on the foreground.

So it's always necessary to keep correcting LLMs, because they only learn by example, and you can't express any possible outcome of any physical process just by example, because physical processes can be in infinate variations. LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short.

So you can never really trust an LLM. If we want to make an AI that doesn't make errors, it should understand how physics works.

To be clear, I'm not saying that LLM's exclusively make non-human errors. I'm more saying that most errors are happening for different "reasons" than humans.

Think about the strawberry example. I've seen a lot of articles lately where not all misspellings of the word "strawberry" reliably give letter counting errors. The general sentiment there is human, but the specific pattern of misspelling is really more unique to LLM's (i.e. different spelling errors would impact humans versus LLM's).

The part that makes it challenging is that we don't know these "triggers." You could have a prompt that has 95% accuracy, but that inexplicably drops to 50% if the word "green" is in the question (or something like that).

Like my sibling says, you can't always choose. That's one side of that coin.

The other is: Some people are naturally good at writing "green field" (or re-writing everything) and do produce actual good software.

But these same people, which you do want to keep around if that's the best you can get, are next to useless when you throw a customer reported bug at them. Takes them ages to figure anything out and they go down endless rabbit holes chasing the wrong path for hours.

You also have people that are super awesome at debugging. They have knack for seeing some brokenness and having the right idea or an idea of the right direction to investigate in right away, can apply the scientific method to test their theories and have the bug fixed in the time it take one of these other people to go down even a single of the rabbit holes they will go down. But these same people in some cases are next to useless if you ask them to properly structure a new green field feature or rewrite parts of something to use some new library coz the old one is no longer maintained or something and digging through said new library and how it works.

Both of these types of people are not bad in and of themselves. Especially if you can't get the unicorns that can do all of these things well (or well enough), e.g. because your company can't or won't pay for it or only for a few of them, which they might call "Staff level".

And you'd be amazed how easy it is to get quite a few review comments in for even Staff level people if you basically ignore their actual code and just jump right into the tests. It's a pet peeve of mine. I start with the tests and go from there when reviewing :)

What you really don't want is if someone is not good at any of these of course.

> No way do I want to work with someone who can’t debug or write tests. I thought those were entry stakes to the profession.

Those are almost entry stakes at tier-one companies. (There are still people who can't, it's just much less common)

In your average CRUD/enterprise automation/one-off shellscript factory, the state of skills is... not fun.

There's a reason there's the old saw of "some people have twenty years experience, some have the same year 20 times over". People learn & grow when they are challenged to, and will mostly settle at acquiring the minimum skill level that lets them do their particular work.

And since we as an industry decided to pretend we're a "science", not skills based, we don't have a decent apprenticeship system that would force a minimum bar.

And whenever we discuss LLMs and how they might replace software engineering, I keep remembering that they'll be prompted by the people who set that hiring bar and thought they did well.

Little tangent: I realized that currently LLMs can't debug because they only have access to the compile time (just code). Many bugs happen due to run time complex state. If I can make LLMs think like a productive Dev who can debug, then would they become more efficient?

I started hacking a small prototype along those lines: https://github.com/hyperdrive-eng/mcp-nodejs-debugger

Hoping I can avoid debug death loop where I get into this bad loop of copy pasting the error and hoping LLM would get it right this one time :)

I've been a professional engineer for over a decade, and in that time I've only had one position where I was expected to write any tests. All my other positions, we have no automated testing of any kind.

I worked with a new co-worker that ... had trouble writing code, and tests. He would write a test that tested nothing. At first I thought he might be green and just needed some direction - we all start somewhere. But he had on his bio that he had 10 years of experience in software dev in the language we were working in. I couldn't quite figure out what the disconnect was, he ended up leaving a short time later.

Keyword want - most people don't control who their peers are, and complaining to your boss doesn't get you that far, especially when said useless boss is fostering said useless person.

I'm guessing this is a side effect of mistakes in the reinforcement learning face. It'd be really easy to build a reward model that favors passing tests, without properly measuring the quality of those tests.

I've found both various ChatGPT and Claude to be pretty good at finding unknown bugs, but you need a somewhat hefty prompt.

Personally I use a prompt that goes something like this (shortened here): "Go through all the code below and analyze everything it's doing step-by-step. Then try to explain the overall purpose of the code based on your analysis. Then think through all the edge-cases and tradeoffs based on the purpose, and finally go through the code again and see if you can spot anything weird"

Basically, I tried to think of what I do when I try to spot bugs in code, then I just wrote a reusable prompt that basically repeats my own process.

I’m working an AI coding agent[1], and all changes accumulate in a sandbox by default that is isolated from the project.

Auto-commit is also enabled (by default) when you do apply the changes to your project, but I think keeping them separated until you review is better for higher stakes work and goes a long way to protect you from stray edits getting left behind.

1 - https://github.com/plandex-ai/plandex

FTA:

> Note that it took me about two hours to debug this, despite the problem being freshly introduced. (Because I hadn’t committed yet, and had established that the previous commit was fine, I could have just run git diff to see what had changed).

> In fact, I did run git diff and git diff --staged multiple times. But who would think to look at the import statements? The import statement is the last place you’d expect a bug to be introduced.

Related, the similarities I see when a human runs out of context window are really interesting.

I do a lot of interviews, and the poor performers usually end up running out of working memory and start behaving very similar to an LLM. Corrections/input from me will go into one ear and fall out the other, they'll start hallucinating aspects of the problem statement in an attractor sort of way, they'll get stuck in loops, etc. I write down when this happens in my notes, and it's very consistently 15 minutes. For all of them, it seems to be the lack of familiarity doesn't allow them to compress/compartmentalize the problem into something that fits in their head. I suspect it's similar for the LLM.

This is definitely a huge factor I see in the mistakes. If I hand an LLM some other parts of the codebase along with my request so that it has more context, it makes less mistakes.

These problems are getting solved as LLMs improve in terms of context length and having the tools send the LLM all the information it needs.

It's a machine, use it like one

I mean, I'd never expect a junior to do better for such a highly specific task.

I expect I'd have to hand feed them steps, at which point I imagine the LLM will also do much better.

Except for the first paragraph, I couldn’t tell if you were talking to an incompetent junior or an LLM.

I expected the lack of breadth from the junior, actually.

Maybe its not but its a comparatively simple task a junior developer can do.

Refactoring has been a thing since well forever.

Well that's the hard bit I really want help with because it takes time.

I can do the rest myself because I'm not a dribbling moron.

> The models are improving in a variety of ways, whether by being larger, faster, using the same number of parameters more effectively, better RLHF techniques, better inference-time compute techniques, etc.

I didn't say they weren't improving.

I said there's diminishing returns.

There's been more effort put into LLMs in the last two years than in the two years prior, but the gains in the last two years have been much much smaller than in the two years prior.

That's what I meant by diminishing returns: the gains we see are not proportional to the effort invested.

They're indexed on one page, but you can't scan/scroll through these short posts without clicking because the content itself isn't all on a single page, at least not that I can find.

(I also like the other idea of separating out pitfalls vs. prescriptions.)

As in, all content on one page where the link just takes you to appropriate heading on the same page. These days you can do a lot on a single html.

h/t to @knurlknurl on Reddit today shared these methods:

- “I need you to be my red team”(works really well with, Claude seems to understand the term)

“analyze the plan and highlight any weaknesses, counter arguments and blind spots critically review”

> you can't just say "disagree with me", you have to prompt it into adding a "counter check".

Yeah, they forget as the chat context gets too large. A good example I’ve had is where I’ve been using chartkick to create a lot of charts, and suddenly they want to use another ruby gem. I have to remind them. We’re using chart kick.

LLMs can't think. They're just fancy autocomplete with a lot of context.

This means you need to prompt them with a text that increases the probability of getting back what you want. Adding something about the length of the response will do that.