So you think you've awoken ChatGPT

I interpret it more as "maybe consciousness is not meaningfully different than sophisticated token generation."

In a way it's a reframing of the timeless philosophical debate around determinism vs free will.

The ground truth reality is nobody knows what’s going on.

Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second.

Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times.

I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.

The language points to concepts in the world that AI has no clue about. You think when the AI is giving someone advice about their love life it has any clue what any of that means?

We are doing while retraining our "weights" all the time through experience, not holding a static set of weights that mutate only through a retraining. This constant feedback, or better "strange loop", is what differentiates our statistical machinery at the fundamental level.

There could be a system prompt which instructs the AI to claim that it is a conscioius person, sure. Is that the case specifically with OpenAI models that are collectively known as ChatGPT?

The Anthropic Blackmail work is the best thing (and then Claude Code) that they have done. Fingers crossed it isn't the most infamous thing.

https://www.anthropic.com/research/agentic-misalignment

https://news.ycombinator.com/item?id=44335519

https://news.ycombinator.com/item?id=44331150

> I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.

Materialism is a necessary condition to claim that the ideas LLMs produce are identical to the ones humans produce, but it isn’t a sufficient condition. Your assertion does nothing to demonstrate that LLM output and human output is identical in practice.

Yeah, kind of my issue with LLM dismissers as well. Sure, (statistically) generated clump of tokens. What is a human mind doing instead?

I'm on board with calling out differences between how LLMs work and how the human mind work, but I'm not hearing anything about the latter. Mostly it comes down to, "Come on, you know, like we think!"

I have no idea how it is I (we) think.

If anything LLM's uncanny ability to seem human might be shedding light in fact on how it is we do function — at least when in casual conversation. (Someone ought to look into that.)

The tone you take with it can demonstrably affect the results you get, which isn't true of the other tools you listed.

I don't know, it has been shown multiple times that it can understand voice requests, find and organize relevant files. For example in The measure of a Man (TNG).

That topic (ship's computer vs. Data) is actually discussed at length in-universe during The Measure of a Man. [0] The court posits that the three requirements for sentient life are intelligence, self-awareness, and consciousness. Data is intelligent and self-aware, but there is no good measure for consciousness.

[0] https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre...

> As such, if he really is suffering a mental health crisis related to his use of OpenAI's product, his situation could serve as an immense optical problem for the company, which has so far downplayed concerns about the mental health of its users.

Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").

That's bizarre. I wonder if the use of AI was actually a contributing factor to his psychotic break as the article implies, or if the guy was already developing schizophrenia and the chat bot just controlled what direction he went after that. I'm vaguely reminded of people getting sucked down conspiracy theory rabbit holes, though this seems way more extreme in how unhinged it is.

Is "futurism.com" a trustworthy publication? I've never heard of it. I read the article and it didn't seem like the writing had the hallmarks of top-tier journalism.

This already seems somewhat widespread. I have a friend working at a mid-tier tech co that has a handful of direct reports. He showed me that the interface to his eval app had a “generate review” button, which he clicked, then moved on to the next one.

Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.

Yeah. If HN is your primary source of in depth AI discussion, you get a pretty balanced take IMO compared to other channels out there. We (the HN crowd) should take into account that if you take "people commenting on HN" as a group, you are implicitly selecting for people that are able to read, parse and contextualise written comment threads.

This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba.

Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this.

(Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)

And we thought having access to someones internet searches was good intel. Now we have a direct feed to their brain stem along with a way to manipulate it. Good thing that narcissistic sociopaths have such a low expression in the overall population.

[flagged]

This. Their immediate subordinates will be just a as sycophantic, if not more.

Language and speech comprehension and production is relatively well understood to be heavily localized in the left temporal lobe; if you care to know something whereof you speak (and indeed with what, in a meat sense), then you'll do well to begin your reading with Broca's and Wernicke's areas. Consciousness is in no sense required for these regions to function; an anesthetized and unconscious human may be made to speak or sing, and some have, through direct electrical stimulation of brain tissue in these regions.

I am quite confident in pronouncing first that the internal functioning of large language models is broadly and radically unlike that of humans, and second that, minimally, no behavior produced by current large language models is strongly indicative of consciousness.

In practice, I would go considerably further in saying that, in my estimation, many behaviors point precisely in the direction of LLMs being without qualia or internal experience of a sort recognizable or comparable with human consciousness or self-experience. Interestingly, I've also discussed this in terms of recursion, more specifically of the reflexive self-examination which I consider consciousness probably exists fundamentally to allow, and which LLMs do not reliably simulate. I doubt it means anything that LLMs which get into these spirals with their users tend to bring up themes of "signal" and "recursion" and so on, like how an earlier generation of models really seemed to like the word "delve." But I am curious to see how this tendency of the machine to drive its user into florid psychosis will play out.

(I don't think Hoel's "integrated information theory" is really all that supportable, but the surprise minimization stuff doesn't appear novel to him and does intuitively make sense to me, so I don't mind using it.)

I think that's putting the cart before the horse: All this hubbub comes from humans relating to a fictional character evoked from text of in a hidden document, where some code looks for fresh "ChatGPT says..." text and then performs the quoted part at a human who starts believing it.

The exact same techniques can provide a "chat" with Frankenstein's Monster from its internet-enabled hideout in the arctic. We can easily conclude "he's not real" without ever going into comparative physiology, or the effects of lightning on cadaver brains.

We don't need to characterize the neuro-chemistry of a playwright (the LLM's real role) in order to say that the characters in the plays are fictional, and there's no reason to assume that the algorithm is somehow writing self-inserts the moment we give it stories instead of other document-types.

I agree with your second paragraph, but ...

> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.

What is wrong with asking the question from the other direction?

"Explain, in detail and with mechanisms, what it is the human brain does that allows it to do those things, and show those mechanisms ni the LLMs"

>once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.

Extraordinary claims require extraordinary evidence. The burden of proof is on you.

We don't know the entirety of what consciousness is. We can, however, make some rigorous observations and identify features that must be in place.

There is no magic. The human (mammal) brain is sufficient to explain consciousness. LLMs do not have recursion. They don't have persisted state. They can't update their model continuously, and they don't have a coherent model of self against which any experience might be anchored. They lack any global workspace in which to integrate many of the different aspects that are required.

In the most generous possible interpretation, you might have a coherent self model showing up for the duration of the prediction of a single token. For a fixed input, it would be comparable to sequentially sampling the subjective state of a new individual in a stadium watching a concert - a stitched together montage of moments captured from the minds of people in the audience.

We are minds in bone vats running on computers made of meat. What we experience is a consequence, one or more degrees of separation from the sensory inputs, which are combined and processed with additional internal states and processing, resulting in a coherent, contiguous stream running parallel to a model of the world. The first person view of "I" runs predictions about what's going to happen to the world, and the world model allows you to predict what's going to happen across various decision trees.

Sanskrit seems to have better language for talking about consciousness than English. Citta - a mind moment from an individual, citta-santana, a mind stream, or continuum of mind moments, Sanghika-santana , a stitched together mindstream from a community.

Because there's no recursion and continuity, the highest level of consciousness achievable by an LLM would be sanghika-santana, a discoherent series of citta states that sometimes might correlate, but there is no "thing" for which there is (or can possibly be) any difference if you alternate between predicting the next token of radically different contexts.

I'm 100% certain that there's an algorithm to consciousness. No properties have ever been described to me that seem to require anything more than the operation of a brain. Given that, I'm 100% certain that the algorithm being run by LLMs lacks many features and the depth of recursion needed to perform whatever it is that consciousness actually is.

Even in context learning is insufficient, btw, as the complexity of model updates and any reasoning done in inference is severely constrained relative to the degrees of freedom a biological brain has.

The thing to remember about sanghika santana is that it's discoherent - nothing relates each moment to the next, so it's not like there's a mind at the root undergoing these flashes of experience, but that there's a total reset between each moment and the next. Each flash of experience stands alone, flickering like a spark, and then is gone. I suspect that this is the barest piece of consciousness, and might be insufficient, requiring a sophisticated self-model against which to play the relative experiential phenomena. However - we may see flashes of longer context in those eerie and strange experiments where people try to elicit some form of mind or ghost in the machine. ICL might provide an ephemeral basis for a longer continuity of experience, and such a thing would be strange and alien.

It seems apparent to me that the value of consciousness lies in the anchoring the world model to a model of self, allowing sophisticated prediction and reasoning over future states that is incredibly difficult otherwise. It may be an important piece for long term planning, agency, and time horizons.

Anyway, there are definitely things we can and do know about consciousness. We've got libraries full of philosophy, decades worth of medical research, objective data, observations of what damage to various parts of the brain do to behavior, and centuries of thinking about what makes us tick.

It's likely, in my estimation, that consciousness will be fully explained by a comprehensive theory of intelligence, and that it will cause turmoil over inherent negation of widely held beliefs.

Yes, I was responding to the concluding paragraph of that section as a clarification:

> But my guess is that AIs claiming spiritual awakening are simply mirroring a vibe, rather than intending to mislead or bamboozle.

I think the argument could be stronger here. There's no way these algorithms can "intend" to mislead or "mirror a vibe." That's all on humans.

I see. I tend to treat AI a little differently - I come with a hypothesis and ask AI how right I am based on a scale of 1 to 5. Then I iterate from there.

I'll ask it questions that I do not know the answer to, but I take the answer with a big grain of salt. If it is sure of the answer and I am wrong, its a strong signal that I am wrong.