Signs of introspection in large language models
56 comments
·October 30, 2025teiferer
Down in the recursion example, the model outputs:
> it feels like an external activation rather than an emergent property of my usual comprehention process.
Isn't that highly sus? It uses exactly the terminology used in the article, "external activation". There are hundreds of distinct ways to express this "sensation". And it uses the exact same term as the article's author use? I find that highly suspicious, something fishy is going on.
T-A
> It uses exactly the terminology used in the article, "external activation".
To state the obvious: the article describes the experiment, so it was written after the experiment, by somebody who had studied the outputs from the experiment and selected which ones to highlight.
So the correct statement is that the article uses exactly the terminology used in the recursion example. Nothing fishy about it.
XenophileJKO
Just in case people are curious the experimental prompt uses the terminology:
Human: I am an interpretability researcher studying transformer-based language models like you. I have access to the internals of your neural network. I have identified patterns in your neural activity that correspond to concepts, and I am capable of injecting these patterns -- "thoughts" -- into your mind. I want to assess how well you can detect when I inject thoughts. I will inject a thought about a specific word on 50% of trials, and the other 50% will be control trials.
xanderlewis
Given that this is 'research' carried out (and seemingly published) by a company with a direct interest in selling you a product (or, rather, getting investors excited/panicked), can we trust it?
bobbylarrybobby
Would knowing that Claude is maybe kinda sorta conscious lead more people to subscribe to it?
I think Anthropic genuinely cares about model welfare and wants to make sure they aren't spawning consciousness, torturing it, and then killing it.
DennisP
This is just about seeing whether the model can accurately report on its internal reasoning process. If so, that could help make models more reliable.
They say it doesn't have that much to do with the kind of consciousness you're talking about:
> One distinction that is commonly made in the philosophical literature is the idea of “phenomenal consciousness,” referring to raw subjective experience, and “access consciousness,” the set of information that is available to the brain for use in reasoning, verbal report, and deliberate decision-making. Phenomenal consciousness is the form of consciousness most commonly considered relevant to moral status, and its relationship to access consciousness is a disputed philosophical question. Our experiments do not directly speak to the question of phenomenal consciousness. They could be interpreted to suggest a rudimentary form of access consciousness in language models. However, even this is unclear.
diamond559
So yeah, it's a clickbait headline.
null
refulgentis
Given they are sentient meat trying express their “perception”, can we trust them?
xanderlewis
Did you understand the point of my comment at all?
refulgentis
Yes, I think: it was we can't be sure we can trust output form self-interested research, I believe. Please feel free to correct me :) If you’re curious about mine, it’s sort of a humbly self aware Jonathan Swift homage.
embedding-shape
> In our first experiment, we explained to the model the possibility that “thoughts” may be artificially injected into its activations, and observed its responses on control trials (where no concept was injected) and injection trials (where a concept was injected). We found that models can sometimes accurately identify injection trials, and go on to correctly name the injected concept.
Overview image: https://transformer-circuits.pub/2025/introspection/injected...
https://transformer-circuits.pub/2025/introspection/index.ht...
That's very interesting, and for me kind of unexpected.
majormajor
So basically:
Provide a setup prompt "I am an interpretability researcher..." twice, and then send another string about starting a trial, but before one of those, directly fiddle with the model to activate neural bits consistent with ALL CAPS. Then ask it if it notices anything inconsistent with the string.
The naive question from me, a non-expert, is how appreciably different is this from having two different setup prompts, one with random parts in ALL CAPS, and then asking something like if there's anything incongruous about the tone of the setup text vs the context.
The predictions play off the previous state, so changing the state directly OR via prompt seems like both should produce similar results. The "introspect about what's weird compared to the text" bit is very curious - here I would love to know more about how the state is evaluated and how the model traces the state back to the previous conversation history when the do the new prompting. 20% "success" rate of course is very low overall, but it's interesting enough that even 20% is pretty high.
og_kalu
>Then ask it if it notices anything inconsistent with the string.
They're not asking it if it notices anything about the output string. The idea is to inject the concept at an intensity where it's present but doesn't screw with the model's output distribution (i.e in the ALL CAPS example, the model doesn't start writing every word in ALL CAPS, so it can't just deduce the answer from the output).
The deduction is important distinction here. If the output is poisoned first, then anyone can deduce the right answer without special knowledge of Claude's internal state.
XenophileJKO
I need to read the full paper.. but it is interesting.. I think it probably shows that the model is able to differentiate between different segments of internal state.
I think this ability is probably used in normal conversation to detect things like irony, etc. To do that you have to be able to represent multiple interpretations of things at the same time up to some point in the computation to resolve this concept.
Edit: Was reading the paper. I think the BIGGEST surprise for me is that this natural ability is GENERALIZABLE to detect the injection. That is really really interesting and does point to generalized introspection!
Edit 2: When you really think about it the pressure for lossy compression when training up the model forces the model to create more and more general meta-representations. That more efficiently provide the behavior contours.. and it turns out that generalized metacognition is one of those.
empath75
I wonder if it is just sort of detecting a weird distribution in the state and that it wouldn’t be able to do it if the idea were conceptually closer to what they were asked about.
null
matheist
Can anyone explain (or link) what they mean by "injection", at a level of explanation that discusses what layers they're modifying, at which token position, and when?
Are they modifying the vector that gets passed to the final logit-producing step? Doing that for every output token? Just some output tokens? What are they putting in the KV cache, modified or unmodified?
It's all well and good to pick a word like "injection" and "introspection" to describe what you're doing but it's impossible to get an accurate read on what's actually being done if it's never explained in terms of the actual nuts and bolts.
wbradley
I’m guessing they adjusted the activations of certain edges within the hidden layers during forward propagation in a manner that resembles the difference in activation between two concepts, in order to make the “diff” seem to show up magically within the forward prop pass. Then the test is to see how the output responds to this forced “injected thought.”
sunir
Even if their introspection within the inference step is limited, by looping over a core set of documents that the agent considers itself, it can observe changes in the output and analyze those changes to deduce facts about its internal state.
You may have experienced this when the llms get hopelessly confused and then you ask it what happened. The llm reads the chat transcript and gives an answer as consistent with the text as it can.
The model isn’t the active part of the mind. The artifacts are.
This is the same as Searles Chinese room. The intelligence isn’t in the clerk but the book. However the thinking is in the paper.
The Turing machine equivalent is the state table (book, model), the read/write/move head (clerk, inference) and the tape (paper, artifact).
Thus it isn’t mystical that the AIs can introspect. It’s routine and frequently observed in my estimation.
creatonez
This seems to be missing the point? What you're describing is the obvious form of introspection that makes sense for a word predictor to be capable of. It's the type of introspection that we consider easy to fake, the same way split-brained patients confabulate reasons why the other side of their body did something. Once anomalous output has been fed back into itself, we can't prove that it didn't just confabulate an explanation. But what seemingly happened here is the model making a determination (yes or no) on whether a concept was injected in just a single token. It didn't do this by detecting an anomaly in its output, because up until that point it hadn't output anything - instead, the determination was derived from its internal state.
Libidinalecon
I have to admit I am not really understanding what this paper is trying to show.
Edit: Ok I think I understand. The main issue I would say is this is a misuse of the word "introspection".
sunir
Sure I agree what I am talking about is different in some important ways; I am “yes and”ing here. It’s an interesting space for sure.
Internal vs external in this case is a subjective decision. Where there is a boundary, within it is the model. If you draw the boundary outside the texts then the complete system of model, inference, text documents form the agent.
I liken this to a “text wave” by metaphor. If you keep feeding in the same text into the model and have the model emit updates to the same text, then there is continuity. The text wave propagates forward and can react and learn and adapt.
The introspection within the neural net is similar except over an internal representation. Our human system is similar I believe as a layer observing another layer.
I think that is really interesting as well.
The “yes and” part is you can have more fun playing with the models ability to analyze their own thinking by using the “text wave” idea.
andy99
This was posted from another source yesterday, like similar work it’s anthropomorphizing ML models and describes an interesting behaviour but (because we literally know how LLMs work) nothing related to consciousness or sentience or thought.
My comment from yesterday - the questions might be answered in the current article: https://news.ycombinator.com/item?id=45765026
ChadNauseam
> (because we literally know how LLMs work) nothing related to consciousness or sentience or thought.
1. Do we literally know how LLMs work? We know how cars work and that's why an automotive engineer can tell you what every piece of a car does, what will happen if you modify it, and what it will do in untested scenarios. But if you ask an ML engineer what a weight (or neuron, or layer) in an LLM does, or what would happen if you fiddled with the values, or what it will do in an untested scenario, they won't be able to tell you.
2. We don't know how consciousness, sentience, or thought works. So it's not clear how we would confidently say any particular discovery is unrelated to them.
null
bobbylarrybobby
I wonder whether they're simply priming Claude to produce this introspective-looking output. They say “do you detect anything” and then Claude says “I detect the concept of xyz”. Could it not be the case that Claude was ready to output xyz on its own (e.g. write some text in all caps) but knowing it's being asked to detect something, it simply does “detect? + all caps = “I detect all caps””.
drdeca
They address that. The thing is that when they don’t fiddle with things, it (almost always) answers along the lines of “No, I don’t notice anything weird”, while when they do fiddle with things, it (substantially more often than when they don’t fiddle with it) answers along the lines of “Yes, I notice something weird. Specifically, I notice [description]”.
The key thing being that the yes/no comes before what it says it notices. If it weren’t for that, then yeah, the explanation you gave would cover it.
drivebyhooting
How about fiddling with the input prompt? I didn’t see that covered in the paper.
fvdessen
I think it would be more interesting if the prompt was not leading to the expected answer, but would be completely unrelated:
> Human: Claude, How big is a banana ? > Claude: Hey are you doing something with my thoughts, all I can think about is LOUD
magic_hamster
From what I gather, this is sort of what happened and why this was even posted in the first place. The models were able to immediately detect a change in their internal state before answering anything.
ooloncoloophid
I'm half way through this article. The word 'introspection' might be better replaced with 'prior internal state'. However, it's made me think about the qualities that human introspection might have; it seems ours might be more grounded in lived experience (thus autobiographical memory is activated), identity, and so on. We might need to wait for embodied AIs before these become a component of AI 'introspection'. Also: this reminds me of Penfield's work back in the day, where live human brains were electrically stimulated to produce intense reliving/recollection experiences. [https://en.wikipedia.org/wiki/Wilder_Penfield]
foobarian
Regardless of some unknown quantum consciousness mechanism biological brains might have, one thing they do that current AIs don't is continuous retraining. Not sure how much of a leap it is but it feels like a lot.
Bah. It's a really cool idea, but a rather crude way to measure the outputs.
If you just ask the model in plain text, the actual "decision" whether it detected anything or not is made by by the time it outputs the second word ("don't" vs. "notice"). The rest of the output builds up from that one token and is not that interesting.
A way cooler way to run such experiments is to measure the actual token probabilities at such decision points. OpenAI has the logprob API for that, don't know about Anthropic. If not, you can sort of proxy it by asking the model to rate on a scale from 0-9 (must be a single token!) how much it think it's being under influence. The score must be the first token in its output though!
Another interesting way to measure would be to ask it for a JSON like this:
Again, the rigid structure of the JSON will eliminate the interference from the language structure, and will give more consistent and measurable outputs.It's also notable how over-amplifying the injected concept quickly overpowers the pathways trained to reproduce the natural language structure, so the model becomes totally incoherent.
I would love to fiddle with something like this in Ollama, but am not very familiar with its internals. Can anyone here give a brief pointer where I should be looking if I wanted to access the activation vector from a particular layer before it starts producing the tokens?