An LLM is a lossy encyclopedia

> the user will at least need to know something about the topic beforehand.

I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]"

It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly corrected itself and provided the correct dosing guidelines.

These are the kind of innocent errors that can be dangerous if users trust it blindly.

The main challenge is LLMs aren't able to gauge confidence in its answers, so it can't adjust how confidently it communicates information back to you. It's like compressing a photo, and a photographer wrongly saying "here's the best quality image I have!" - do you trust the photographer at their word, or do you challenge him to find a better quality image?

> the user will at least need to know something about the topic beforehand.

This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answer as to why that code is awful.

I use it mostly on personal projects especially since I can prototype quickly as needed.

>the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input

I think there's a parallel here for the internet as an i formation source. It delivered on "unlimited knowledge at the tip of everyone's fingertips" but lowering the bar also lowered the bar.

That access "works" only when the user is capable of doing their part too. Evaluating sources, integrating knowledge. Validating. Cross examining.

Now we are just more used to recognizing that accessibility comes with its own problem.

Some of this is down to general education. Some to domain expertize. Personality plays a big part.

The biggest factor is, i think, intelligence. There's a lot of 2nd and 3rd order thinking required to simultaneously entertain a curiosity, consider of how the LLM works, and exercise different levels of skepticism depending on the types of errors LLMs are likely to make.

Using LLMs correctly and incorrectly is.. subtle.

> The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters

It's also useful to have an intuition for what things an LLM is liable to get wrong/hallucinate, one of which is questions where the question itself suggests one or more obvious answers (which may or may not be correct), which the LLM may well then hallucinate, and sound reasonable, if it doesn't "know".

https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

> the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand

This is why simonw (The author) has his "pelican on a bike" -test, it's not 100% accurate but it is a good indicator.

I have a set of my own standard queries and problems (no counting characters or algebra crap) I feed to new LLMs I'm testing

None of the questions exist outside of my own Obsidian note so they can't be gamed by LLM authors. And I've tested multiple different LLMs using them so I have a "feeling" on what the answer should look like. And I personally know the correct answer so I can immediately validate them.

I get very annoyed when llms respond with quotes around certain things I ask for, then when I say what is the source of that quote? they say oh I was paraphrasing and that isnt a real quote.

At least wikipedia has sources that probably support what it says and normally the quotes are real quotes. LLMs just seem to add quotation marks as, "proof" that its confident something is correct.

To that end, it seems as though archive.org will important for an entirely new reason. Not for the loss of information, but the degradation of it.

[flagged]

Lossy compression does make things up. We call them compression artefacts.

In compressed audio these can be things like clicks and boings and echoes and pre-echoes. In compressed images they can be ripply effects near edges, banding in smoothly varying regions, but there are also things like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... where one digit is replaced with a nice clean version of a different digit, which is pretty on-the-nose for the LLM failure mode you're talking about.

Compression artefacts generally affect small parts of the image or audio or video rather than replacing the whole thing -- but in the analogy, "the whole thing" is an encyclopaedia and the artefacts are affecting little bits of that.

Of course the analogy isn't exact. That would be why S.W. opens his post by saying "Since I love collecting questionable analogies for LLMs,".

I don't think there is a singular "should" that fits every use case.

E.g. a Bloom filter also doesn't "know" what it knows.

The argument is that a banana is a squishy hammer.

You're saying hammers shouldn't be squishy.

Simon is saying don't use a banana as a hammer.

Yeah an LLM is an unreliable librarian, if anything.

I actually disagree. Modern encoding formats can, and do, hallucinate blocks.

It’s a lot less visible and I guess dramatic than LLMs but it happens frequently enough that I feel like at every major event there are false conspiracies based on video « proofs » that are just encoding artifacts

I think you are missing the point of the analogy: a lossy encyclopedia is obviously a bad idea, because encyclopedias are meant to be reliable places to look up facts.

> You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt.

Oh but it's much worse than that: because most LLMs aren't deterministic in the way they operate [1], you can get a pristine image of a different pile of dirt every single time you ask.

[1] there are models where if you have the "model + prompt + seed" you're at least guaranteed to get the same output every single time. FWIW I use LLMs but I cannot integrate them in anything I produce when what they output ain't deterministic.

An encyclopaedia also can't win a gold medals at the IMO and IOI. So yeah, they're not the same thing, even though the analogy is pretty good.

Using artificial neural networks directly for information storage and retrieval (i.e. not just leveraging them as tools accessing other types of storage) is currently infeasible, agreed.

On the other hand, biological neural networks are doing it all the time :) And there might well be an advantage to it (or a hybrid method), once we can make it more economical.

After all, the embedding vector space is shaped by the distribution of training data, and if you have out-of-distribution data coming in due to a new or changed environment, RAG using pre-trained models and their vector spaces will only go so far.

Are LLMs really lossier than humans? I think it depends on the context. Given any particular example, LLMs might hallucinate more and a human might do a better job at accuracy. But overall LLMs will remember far more things than a human. Ask a human to reproduce what they read in a book last year and there's a good chance you'll get either absolutely nothing or just a vague idea of what the book was about - in this context they can be up to 100% lossy. The difference here is that human memory decays over time while a LLM's memory is hardwired.

Lossy is an incomplete characterization. LLMs are also much more fluctuating and fuzzy. You can get wildly varying output depending on prompting, for what should be the same (even if lossy) knowledge. There is not just loss during the training, but also loss and variation during inference. An LLM overall is a much less coherent and consistent thing than most humans, in terms of knowledge, mindset, and elucidations.

> If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordingly and survive.

Yes.

Human intelligence consists of three things.

First, groundedness: The ability to form a representation of the world and one’s place in it.

Second, a temporal-spatial sense: A subjective and bounded idea of self in objective space and time.

Third: A general predictive function which is capable of broad abstraction.

At its most basic level, this third element enables man to acquire, process, store, represent, and continually re-acquire knowledge which is external to that man's subjective existence. This is calculation in the strictest sense.

And it is the third element -- the strength, speed, and breadth of the predictive function -- which is synonymous with the word "intelligence." Higher animals have all three elements, but they're pretty hazy -- especially the third. And, in humans, short time horizons are synonymous with intellectual dullness.

All of this is to say that if you have a "prediction machine" you're 90% of the way to a true "intelligence machine." It also, I think, suggests routes that might lead to more robust AI in the future. (Ground the AI, give it a limited physical presence in time and space, match its clocks to the outside world.)

Imagine having the world's most comprehensive encyclopedia at your literal fingertips, 24 hours a day, but being so lazy that you offload the hard work of thinking by letting retarded software pathologically lie to you and then blindly accepting the non-answers it spits at you rather than typing in two or three keywords to Wikipedia and skimming the top paragraph.

>I prefer to think of LLMs as lossy predictors.

I've started to call them the Great Filter.

In the latest issue of the comic book Lex Luthor attempts to exterminate humanity by hacking the LLM and having it inform humanity that they can hold their breath underwater for 17 hours.

> you build a world model

The foundational conceit (if you will) of LLMs is that they build a semantic (world) model to 'make sense' of their training. However it is much more likely that they are simply building a syntactic model in response to the training. As far as I know there is no evidence of a semantic model emerging.

Another difference is that you are predicting future sensory experiences in real-time, while LLMs "predict" text which a "helpful, honest, harmless" assistant would produce.

I suspect this ends up being pretty important for the next advancements in AI, specifically LLM-based AI. To me, the transformer architecture is a sort of compression algorithm that is being exploited for emergent behavior at the margins. But I think this is more like stream of consciousness than premeditated thought. Eventually I think we figure out a way to "think" in latent space and have our existing AI models be just the mouthpiece.

In my experience as a human, the more you know about a subject, or even the more you have simply seen content about it, the easier it is to ramble on about it convincingly. It's like a mirroring skill, and it does not actually mean you understand what you're saying.

LLMs seem to do the same thing, I think. At scale this is widely useful, though, I am not discounting it. Just think it's an order of magnitude below what's possible and all this talk of existing stream-of-consciousness-like LLMs creating AGI seems like a miss

One difference is that compression gives you one and only one thing when decompressing. Decompression isn't a function taking arbitrary additional input and producing potentially arbitrary, nondeterministic output based on it.

We would have very different conversations if LLMs were things that merely exploded into a singular lossy-expanded version of Wikipedia, but where looking at the article for any topic X would give you the exact same article each time.

With a handy trick called arithmetic coding, you can actually turn an LLM into a lossless compression algorithm!

Worth highlighting - 2023.

I don't think that's universally true with the new models - I've seen Claude 4 and GPT-5 ask for clarification on questions with obvious gaps.

With GPT-5 I sometimes see it spot a question that needs clarifying in its thinking trace, then pick the most likely answer, then spit out an answer later that says "assuming you meant X ..." - I've even had it provide an answer in two sections for each branch of a clear ambiguity.

This is also why the Kagi Assistant is still be the AI tool I’ve found. The failure state is the same as a search results, it either can’t find anything, finds something irrelevant, or finds material that contradicts the premise of your question.

It seems to me the more you can pin it to another data set, the better.

It's a version of the library of babel filtered only to the books which plausibly consist of prose. The set is still incomprehensibly vast, and all the more treacherous for generally being "readable".