Life is more than an engineering problem
256 comments
·February 2, 2025abecedarius
weego
Counter point: what is it about scraping the Internet and indexing it cleverly that makes you believe that would lead to the the creation of the ability to reason above it's programming?
No one in neuroscience, psychology or any related field can point to reasoning or 'consciousness' or whatever you wish to call it and say it appeared from X. Yet we have this West Coast IT cultish thinking that if we throw money at it we'll just spontaneously get there. The idea that we're even 1% close should be ridiculous to anyone rationally looking at what we're currently doing.
abdullahkhalids
> No one in neuroscience, psychology or any related field can point to reasoning or 'consciousness' or whatever you wish to call it and say it appeared from X.
This is not a good argument. Natural systems, the subject of neuroscience/psychology, are much harder to analyze than artificial systems. For example, it's really difficult to study atmospheric gases and figure out Boyle's/Charles law. But put a gas in a closed chamber and change pressure or temperature and these laws are trivially apparent.
LLMs are much more legible systems than animal brains, and they are amenable to experiment. So, it is much more likely that we will be able to identify what "reasoning" is by studying these systems than animal brains.
P.S. Don't think we are there yet, as much as internet commentators might assert.
hnisoss
Yea but following your example/analogy you have gas-gas but brain-llm. So how can we then experiment? It's a simulation at best.
afpx
This paper may be interesting to some of you:
Discretization of continuous input spaces in the hippocampal autoencoder
wholinator2
I think it's really up to the highly nebulous definition. Even in your comment is implied that reasoning and consciousness are two names of the same thing but i'd argue one is here and one will never be provable ever. Reason is working through logical steps, much like a program. It's a set of conditions that get checked and a logical structure that uses that information to reach a conclusion. That's what sets it apart from gut feelings or emotional thinking, it's a traceable structure with "reasons". I can watch the LLM speak base facts out loud, then begin to synthesize them giving _reasons_ for the choices it's making, culminating in a final conclusion. It's already doing that. That is what i call reason. It doesn't mean it's human, it doesn't mean it's "aware of itself", it just means it's thinking a train of thought with concrete steps between each car. Consciousness is completely undefinable and useless as a metric and will never be probably achieved.
bsenftner
I agree that reasoning and consciousness are different, however what I do not see being discussed by the AI research community is the necessity to define and then develop "artificial comprehension".
At this point in time, the act of comprehension is a scientific mystery.
I'd say 'consciousness' is the ongoing ever present comprehension of the moment, a feedback self conversation assessing the current situation a being finds itself. This act requires reasoning, as comprehension is the "sandbox" in which reasoning occurs.
But what is comprehension? It's the instantaneous reverse engineering of observations for verification of reality: is what I observe normal, possible or a threat? If one cannot "understand" an observation then the potential the observation is a threat grows. That 'understanding" is reverse engineering the observation to identify it's range of possible behavior and therefore one's safety in relation to that observation.
Comprehension is extremely complex: arbitrary input goes in and a world model with one's safety and next actions comes out.
acchow
The assumption is that since there is already a neural network that “got there” (our brains), we should be able to achieve the same thing synthetically.
We just need to figure out how to train that network.
aaarrm
Neural networks are a simplification of our brains, they are not a replication of it. It is just a modeling method that was inspired by how human neurons work, that's it. It's not 1 to 1 or anything.
bjelkeman-again
It may be possible to argue that current work in AI leads to some definition of intelligence, which apparently often is equaled to consciousness by some.
My take it is just unaware intelligence like in Peter Watts’ book Blindsight. A terrific read and a quite scary prospect.
rcxdude
It's more that if you actually work with LLMs they will display reasoning. It's not particularly good or deep reasoning (I would generally say they have a superhuman amount of knowledge but are really quite unintelligent), but it is more than simply recall.
d4mi3n
Waters are often muddied here by our own psychology. We (as a species) tend to ascribe intelligence to things that can speak. Even more so when someone (or thing in this case) can not just speak, but articulate well.
We know these are algorithms, but how many people fall in love or make friends over nothing but a letter or text message?
Capabilities for reasoning aside, we should all be very careful of our perceptions of intelligence based solely on a machines or algorithms apparent ability to communicate.
fatbird
Are they displaying reasoning, or the outcome of reasoning, leading you to a false conclusion?
Personally, I see ChatGPT say "water doesn't freeze at 27 degrees F" and think "how can it possibly do advanced reasoning when it can't do basic reasoning?"
whatshisface
I don't think any of us are qualified to tell the difference between exhibiting reasoning and mixing examples taken from the entire internet. Maybe if the training data was small enough to comprehend in its entirety, we could say one way or the other, but as it stands none of us have read the enitre internet, and we have no way of finding the stackoverflow or Reddit conversation that most closely resembles a given chain of thought.
abecedarius
Yes, my judgement too from messing with Claude and (previously) ChatGPT. 'Ridiculous' and 'cultish' are overton-window enforcement more than they are justified.
nuancebydefault
From its answers I already conclude it is already reasoning above its programming. I do not see why someone in neuroscience or psychology would need to say it appeared, since they do not know better what reasoning is than any average human.
Reasoning is undefined, but a human recognizes it when it appears. I don't see consciousness part of that story. Also, whether you call it emulated or played reasoning or not, apparently does not matter. The results are what they are.
sleepytree
If I write a book that contains Einstein's theory of relativity by virtue of me copying it, did I create the theory? Did my copying of it indicate anything about my understanding of it? Would you be justified to think the next book I write would have anything of original value?
I think what he is trying to say is that LLMs current architecture seems to mainly work by understanding patterns in the existing body of knowledge. In some senses finding patterns could be considered creative and entail reasoning. And that might be the degree to which LLMs could be said to be capable of reasoning or creativity.
But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of.
TeMPOraL
> If I write a book that contains Einstein's theory of relativity by virtue of me copying it, did I create the theory? Did my copying of it indicate anything about my understanding of it? Would you be justified to think the next book I write would have anything of original value?
No, but you described a `cp` command, not an LLM.
"Creativity" in the sense of coming up with something new is trivial to implement in computers, and has long been solved. Take some pattern - of words, of data, of thought. Perturb it randomly. Done. That's creativity.
The part that makes "creativity" in the sense we normally understand it hard, isn't the search for new ideas - it's evaluation of those ideas. For an idea to be considered creative, it has to match a very complex... wait for it... pattern.
That pattern - what we call "creative" - has no strict definition. The idea has to be close enough to something we know, so we can frame it, yet different enough from it as to not be obvious, but still not too different, so we can still comprehend it. It has to make sense in relevant context - e.g. a creative mathematical proof has to still be correct (or a creative approach to proving a theorem has to plausibly look like it could possibly work); creative writing still has to be readable, etc.
The core of creativity is this unspecified pattern that things we consider "creative" match. And it so happens that things matching this pattern are a match for pattern "what makes sense for a human to read" in situations where a creative solution is called for. And the latter pattern - "response has to be sensible to a human" - is exactly what the LLM goal function is.
Thus follows that real creativity is part of what LLMs are being optimized for :).
sleepytree
> For an idea to be considered creative, it has to match a very complex... wait for it... pattern.
If we could predefine what would count as creativity as some specific pattern, then I'm not sure that would be what I would call creative, and certainly wouldn't be an all-inclusive definition of creativity. Nor is creativity merely creating something new by perturbing data randomly as you mentioned above.
While LLMs might be capable of some forms of creativity depending on how you define it, I think it remains to be seen how LLMs' current architecture could on its own accomplish the kinds of creativity implicit in scientific progress in the Kuhnian sense of a paradigm shift or in what some describe as a leap of artistic inspiration. Both of these examples highlight the degree to which creativity could be considered both progress in an objective sense but also be something that is not entirely foreshadowed by its precursors or patterns of existing data.
I think there are many senses in which LLMs are not demonstrating creativity in a way that humans can. I'm not sure how an LLM itself could create something new and valuable if it requires predefining an existing pattern which seems to presuppose that we already have the creation in a sense.
corimaith
>Creativity" in the sense of coming up with something new is trivial to implement in computers, and has long been solved. Take some pattern - of words, of data, of thought. Perturb it randomly. Done. That's creativity.
Formal Proof Systems aren't even nearly close to completion, and for patterns we don't have a strong enough formal system to fully represent the problem space.
If we take the P=NP problem, that likely can be solved formally that a machine could do, but what is the "pattern" here that we are traversing here? There is a definitely a deeper superstructure behind these problems, but we can only glean the tips, and I don't think the LLMs with statistical techniques can glean further in either. Natural Language is not sufficient.
daveguy
> Take some pattern - of words, of data, of thought. Perturb it randomly. Done. That's creativity.
This seems a miopic view of creativity. I think leaving out the pursuit of the implications of that perturbation is leaving out the majority of creativity. A random number generator is not creative without some way to explore the impact of the random number. This is something that LLM inference models just don't do. Feeding previous output into the context of a next "reasoning" step still depends on a static model at the core.
glenstein
>If I write a book that contains Einstein's theory of relativity by virtue of me copying it, did I create the theory? Did my copying of it indicate anything about my understanding of it? Would you be justified to think the next book I write would have anything of original value?
If you, after copying the book, could dynamically answer questions about the theory, it's implications, and answer variations of problems or theoretical challenges in ways that reflect mainstream knowledge, I think that absolutely would indicate understanding of it. I think you are basically making Searle's chinese room argument.
>But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of.
Why is that clear? I think the reasoning for that would be tying it to a notion "the human experience", which I don't think is a necessary condition for intelligence. I think nothing about finding patterns is "mere" insofar as it relates to demonstration of intelligence.
vouwfietsman
> But it is clear humans are capable of ...
Its not though, nobody really knows what most of the words in that sentence mean in the technical or algorithmical sense, and hence you can't really say whether llms do or don't possess these skills.
southernplaces7
>nobody really knows what most of the words in that sentence mean in the technical or algorithmical sense
And nobody really knows what consciousness is, but we all experience it in a distinct, internal way that lets us navigate the world and express ourselves to others, yet apparently some comments seem to dismiss this elephant of sensation in the room by pretending it's no different than some cut and dried computational system that's programmed to answer certain things in certain ways and thus "is probably no different from a person trained to speak". We're obviously, evidentially more than that.
mrcsd
Words are not reducible to technical statements or algorithms. But, even if they were, then by your suggestion there's not much point in talking about anything at all.
Rury
> LLMs current architecture seems to mainly work by understanding patterns in the existing body of knowledge ...
>But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of
This is not clear at all. As it seems to me, it's impossible to imagine or think of things that are not in someway tied to something you've already come to sense or know. And if you think I am wrong, I implore you to provide a notion that doesn’t agree. I can only imagine something utterly unintelligible, and in order to make it intelligible, would require "pattern matching" (ie tying) it to something that is already intelligible. I mean how else do we come to understand a newly-found dead/unknown language, or teach our children? What human thought operates completely outside existing knowledge, if not given empirically?
rileymat2
Why can’t creativity be taking the works, a bunch of works, finding a pattern then randomly perturbing a data point/concept to see if there are new patterns.
Then cross referencing that new random point/idea to see if it remains internally consistent with the known true patterns in your dataset.
This is how humans create new ideas often?
southernplaces7
I see absolutely zero wrong with that statement. What he said is indeed much more reasoned and intelligent than the average foolish AI hype i've often found here, written by people who try to absurdly redefine the obvious, complex mystery that is consciousness into some reductionist notion of it being anything that presents the appearance of reasoning through technical tricks.
Chiang has it exactly right with his doubts, and the notion that pattern recognition is little different from the deeply complex navigation of reality we living things do is the badly misguided notion.
munksbeer
> Chiang has it exactly right with his doubts, and the notion that pattern recognition is little different from the deeply complex navigation of reality we living things do is the badly misguided notion.
How do you know this?
daveguy
The same way that we know interpolation of a linear regression is not the same as the deeply complex navigation of reality we do as living things.
mewpmewp2
Even most intelligent people can hallucinate, we still haven't fixed this problem. There's a lot of training material and bias which leads many to repeat those things "LLM's are just a stochastic parrot, glorified auto complete/google search, Markov chains, just statistics", etc. The thing is, these sentences sound really good and so it's easy to repeat them when you have made up your mind. It's a shortcut.
dutchbookmaker
I feel like at this point we have to separate LLMs and reasoning models too.
I can see the argument against chatGPT4 reasoning.
The reasoning models though I think get into some confusing language but I don't know what else you would call it.
If you say a car is not "running" the way a human runs, you are not incorrect even though a car can "outrun" any human obviously in terms of moving speed on the ground.
To say since a car can't run , it can't move though is obviously completely absurd.
bonoboTP
This was precisely what motivated Turing to come up with the test named after him, to avoid such semantic debates. Yet here we are still in the same loop.
"The terminator isn't really hunting you down, it's just imitating doing so..."
kortilla
LLMs don’t go into a different mode when they are hallucinating. That’s just how they work.
Using the word “hallucinate” is extremely misleading because it’s nothing like what people do when they hallucinate (thinking there are sensory inputs when there aren’t).
It’s much closer to confabulation, which is extremely rare and is usually a result of brain damage.
This is why a big chunk of people (including myself) think the current LLMs are fundamentally flawed. Something with a massive database to statistically confabulate correct stuff 95% of the time and not have a clue when it’s completely made up is not anything like intelligence.
Compressing all of the content of the internet into an LLM is useful and impressive. But these things aren’t going to start doing any meaningful science or even engineering on their own.
acureau
Intelligent people do not "hallucinate" in the same sense that an LLM does. Counterarguments you don't like aren't "shortcuts". There are certainly obnoxious anti-LLM people, but you can't use them to dismiss everyone else.
An LLM does nothing more than predict the next token in a sequence. It is functionally auto-complete. It hallucinates because it has no concept of a fact. It has no "concept", period, it cannot reason. It is a statistical model. The "reasoning" you observe in models like o1 is a neat prompting trick that allows it to generate more context for itself.
I use LLMs on a daily basis. I use them at work and at home, and I feel that they have greatly enhanced my life. At the end of the day they are just another tool. The term "AI" is entirely marketing preying on those who can't be bothered to learn how the technology works.
cyrillite
They’re right until they’re wrong.
AI is (was?) a stochastic parrot. At some point AI will likely be more than that. The tipping point may not be obvious.
compiler_queen
> Even most intelligent people can hallucinate, we still haven't fixed this problem.
No we have not, neurodiverse people like me need accommodations not fixing.
watwut
It is not hallucination. When people do what we call halucination in chatGPT, it is called "bullshiting", "lying" or "being incompetent".
eterps
> and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here?
Large language models excel at processing and generating text, but they fundamentally operate on existing knowledge. Their creativity appears limited to recombining known information in novel ways, rather than generating truly original insights.
True reasoning capability would involve the ability to analyze complex situations and generate entirely new solutions, independent of existing patterns or combinations. This kind of deep reasoning ability seems to be beyond the scope of current language models, as it would require a fundamentally different approach—what we might call a reasoning model. Currently, it's unclear to me whether such models exist or if they could be effectively integrated with large language models.
nuancebydefault
> True reasoning capability would involve the ability to analyze complex situations and generate entirely new solutions, independent of existing patterns or combinations.
You mean like alphago did in its 36th move?
eterps
Isn't that a non-generic 'reasoning-model' instead of something that is reminiscent of the large language model based AIs we use today?
The question is, is it possible to make reasoning models generic and can they be combined with large language models effectively.
optimalsolver
Move 37.
FrustratedMonky
"Their creativity appears limited to recombining known information"
There are some theories that this is true for humans also.
There are no human created images that weren't observed first in nature in some way.
For example, Devils/Demons/Angels were described in terms of human body parts, or 'goats' with horns. Once we got microscopes and started drawing insects then art got a lot weirder, but not before images were observed from reality. Then humans could re-combine them.
eterps
I understand your point, but it's not comparable:
Humans can suddenly "jump" cognitive levels to see higher-order patterns. Gödel seeing that mathematics could describe mathematics itself. This isn't combining existing patterns, but seeing entirely new levels of abstraction.
The human brain excels at taking complex systems and creating simpler mental models. Newton seeing planetary motion and falling apples as the same phenomenon. This compression isn't recombination - it's finding the hidden simplicity.
Recombination adds elements together. Insight often removes elements to reveal core principles. This requires understanding and reasoning.
freejazz
>and wonder how an intelligent person can still think this, can be so absolute about it.
I wonder how people write things like this and don't realize they sound as sanctimonious as exactly whatever they are criticizing. Or, if I was to put it in your words: "how could someone intelligent post like this?"
abecedarius
You're right, it was kind of rude. Apologies. I really would rather be wrong, for a reason I gave in another comment.
The thing is, you can interact with this new kind of actor as much as you need to to judge this -- make up new problems, ask your own questions. "LLMs can't think" has needed ever-escalating standards for "real" thinking over the last few years.
Gary Marcus made a real-money bet about this.
freejazz
I think a better question is "what is the value of thought?" when it came to conclusions such as yours: "I should be rude to this poster because they disagree with me"
mofeien
To me this also feels like a statement that would obviously need strong justification. For if animals are capable of reasoning, probably through being trained on many examples of the laws of nature doing their thing, then why couldn't a statistical model be?
fmbb
> For if animals are capable of reasoning
Are they? Which animals? Some seem smart and maybe do it. Needs strong justification.
> probably through being trained on many examples of the laws of nature doing their thing
Is that how they can reason? Why do you think so? Sounds like something that needs strong justification.
> then why couldn't a statistical model be?
Maybe because that is not how anything in the world attained the ability to reason.
A lot of animals can see. They did not have to train for this. They are born with eyes and a brain.
Humans are born with the ability to recognize pattern in what we see. We can tell objects apart without training.
unification_fan
> Needs strong justification.
if animals didn't show problem-solving skills, and thus reasoning, complex ones wouldn't exist anymore by now. Planning is a fundamental skill for survival in a resource-constrained environment and that's how intelligence evolved to begin with.
Assuming that intelligence and by extension reasoning are discrete steps is so backwards to me. They are quite obviously continuously connected all the way back to the first nervous systems.
ElevenLathe
Are human beings not animals? If animals can't reason, then neither can we.
slibhb
I agree with Chiang. Reminds me of Searle and The Chinese Room (I agree with Searle too).
I do think that at some point everyone is just arguing semantics. Chiang is arguing that "actual reasoning" is, by definition, not something that an LLM can do. And I do think he's right. But the real story is not "LLMs can't do X special thing that only biological life can do," the real story is "X special thing that only biological life can do isn't necessary to build incredibe AI that in many ways surpasses biological life".
ChrisKnott
> "It’s like imagining that a printer could actually feel pain because it can print bumper stickers with the words ‘Baby don’t hurt me’ on them. It doesn’t matter if the next version of the printer can print out those stickers faster, or if it can format the text in bold red capital letters instead of small black ones. Those are indicators that you have a more capable printer but not indicators that it is any closer to actually feeling anything"
Love TC but I don't think this argument holds water. You need to really get into the weeds of what "actually feeling" means.
To use a TC-style example... suppose it's a major political issue in the future about AI-rights and whether AIs "really" think and "really" feel the things they claim. Eventually we invent an fMRI machine and model of the brain that can conclusively explain the difference between what "really" feeling is, and only pretending. We actually know exactly which gene sequence is responsible for real intelligence. Here's the twist... it turns out 20% of humans don't have it. The fake intelligences have lived among us for millennia...!
epr
I disagree. The reason humans anthropomorphize "AI" is because we apply our own meta-models of intelligence to llms, etc., where they simply don't apply. The model can spit out something that seems extremely intelligent and well thought out that would truly be shocking if a monkey said it for example due to our meta-model of intelligence, and that may be valid in that case if we determined it wasn't simply memorized. His argument can certainly be more fleshed out, but the point he's making is correct, which is that we can't treat the output of a machine designed to replicate human input as though it contains the requisite intelligence/"feeling"/etc to produce that output on it's own.
ChrisKnott
I agree that with current LLMs the error goes the other way; they appear more conscious than they are, compared to, say, crows or octopuses which appear less conscious than they actually are.
My point is that "appears conscious" is really the only test there is. In what way is a human that says "that hurts" really feeling pain? What about Stephen Hawking "saying it", what about if he could only communicate through printed paper etc etc. You can always play this dial-down-the-consciousness game.
People used to say fish don't feel pain, they are "merely responding to stimulus".
mewpmewp2
The only actual difference in my view is that somehow we feel that we are so uber special. Besides that, it seems there's no reason to believe that we are anything more than chemical signals. But the fact that we have this strong "feeling" that we are special refuses us to admit that. I feel like I'm special, I feel like I exist. That's the only argument for being more than something else.
pixl97
Hell, people used to say other people of different races don't feel pain, so we're not a great group to ask because of our biases and motivations.
gcanyon
Interestingly the movie Companion, out this weekend, illustrates this case exactly. It's a thriller, not a philosophical treatise, so don't expect it to go deep into the subject, but the question of what "pain" means to an AI is definitely part of the story.
ge96
I like the one with Ryan Gosling more
null
thwackamole
You appear to be conflating 'feeling' and 'intelligence', which is not what TC is doing.
He is also not wrong about whether current AIs experience feelings. I suggest you learn more about the neuroscience of feelings.
ChrisKnott
Well, he is making an analogy that real internal experience cannot be confirmed externally, however convincing the performance, but this is the only way we know about the internal experience of all things, including ones we typically assign "real" consciousness to (humans, dogs) and ones we don't (amoeba, zygotes, LLMs).
To be clear I'm not for a moment suggesting current AIs are remotely comparable to animals.
asdf6969
> You need to really get into the weeds of what "actually feeling" means.
We don’t even know what this means when it’s applied to humans. We could explain what it looks like in the brain but we don’t know what causes the perception itself. Unless you think a perfect digital replica of a brain could have an inner sense of existence
Since we don’t know what “feeling” actually is there’s no evidence either way that a computer can do it. I will never believe it’s possible for an LLM to feel.
layer8
> I will never believe it’s possible for an LLM to feel.
Why is that, given that, as you state, we don’t know what “feeling” actually is?
armchairhacker
“Feeling” is disconnected from reality, it’s whatever you perceive it as. Like morality, you can’t disprove someone’s definition of feeling, you can only disagree with it.
If scientists invent a way to measure “feeling” that states 20% of people don’t feel, including those otherwise indistinguishable from feeling ones, most people would disagree with the measurement. Similarly, most people would disagree that a printer that prints “baby don’t hurt me” is truly in pain.
nonameiguess
These discussions seem to me to still get hung up on the classical sci-fi view of an AI, even talking about Companion here, of some single identifiable discrete entity that can even potentially be the locus of things like rights and feelings.
What is ChatGPT? Ollama? DeepSeek-R1? They're software. Software is a file. It's a sequence of bytes that can be loaded into memory, with the code portion pulled into a processor to tell it what to do. Between instructions, the operating system it runs on context switches it out back to memory, possibly to disk. Possibly it may crash in the middle of an instruction, but if the prior state was stored off somewhere, it can be recovered.
When you interact through a web API, what are you actually interacting with? There may be thousands of servers striped across the planet constantly being brought offline and online for maintenance, upgrades, A/B tests, hardware decommissioning. The fact that the context window and chat history is stored out of band from the software itself provides an illusion that you're talking to some continually existing individual thing, but you're not. Every individual request may be served by a separate ephemeral process that exists long enough to serve that request and then never exists again.
What is doing the "feeling" here? The processor? Whole server? The collection? The entire Internet? When is it feeling? In the 3 out of 30000 time slices per microsecond that the instruction executing is one pulled from ChatGPT and not the 190 other processes running at the same time that weren't created by machine learning and don't produce output that a human would look at and might think a human produced it?
I'll admit that humans are also pretty mysterious if you reduce us to the unit of computation and most of what goes on in the body and brain has nothing to do with either feeling or cognition, but we know at least there is some qualitative, categorical difference at the structural level between us and sponges. We didn't just get a software upgrade. A GPU running ChatGPT, on the other hand, is exactly the same as a GPU running Minecraft. Why would a fMRI looking at one versus the other see a difference? It's executing the same instructions, possibly even acting on virtually if not totally identical byte streams, and it's only at a higher-level step of encoding that an output device interprets one as rasters and one as characters. You could obfuscate the code the way malware does to hide itself, totally changing the magnetic signature, but produce exactly the same output.
Consider where that leads as a thought experiment. Remove the text encodings from all of the computers involved, or just remove all input validation and feed ChatGPT a stream of random bytes. It'll still do the same thing, but it will produce garbage that means nothing. Would you still recognize it as an intelligent, thinking, feeling thing? If a human suffers some injury to eyes and ears, or is sent to a sensory deprivation chamber, we would say yes, they are still a thinking, feeling, intelligent creature. Our ability to produce sound waves that encode information intelligible to others is an important characteristic, but it's not a necessary characteristic. It doesn't define us. In a vacuum as the last person alive with no way to speak and no one to speak to, we'd still be human. In a vacuum as the last server alive with no humans left, ChatGPT would be dirty memory pages never getting used and eventually being written out to disk by its operating system as the server it had been running on performs automated maintenance functions until it hits a scheduled shutdown, runs out of power, or gets thermally throttled by its BIOS because the data center is no longer being actively cooled.
I think Ted Chiang is doing us a service here. Behavioral equivalence with respect to the production of digitally-encoded information is not equivalence. These things are not like us.
drbig
> Behavioral equivalence
It seems for a lot of people that's all that matters: "if it quacks like a duck it must be a duck!". I find that short-sighted at best, but it's always difficult to present arguments that would "resonate" with the other side...
almusdives
> We actually know exactly which gene sequence is responsible for real intelligence.
We don't at all know this.
borski
It’s an idea for a science fiction story.
bee_rider
It isn’t just an idea for a science fiction story though. It is also a philosophical argument, just predicated on something unexpected, which is probably not true, but which presents for us an interesting scenario, and which isn’t technically ruled out by existing evidence (although, it seems unlikely of course).
Well, I guess that’s what the best science fiction stories are. But, the best science fiction stories aren’t just science fiction stories!
jodrellblank
“suppose”
emtel
I like Chiang’s stories, but he simply doesn’t know what he’s talking about when it comes to LLMs. Forget the philosophical points - he claims that LLMs are strictly worse than search engines for information retrieval. This is just obviously false, I can give you dozens of conversations from my ChatGPT history over the past two months which would have been impossible to get answers to on Google.
benreesman
LLMs are generally better search tools for certain topics today, but search engines have been in serious decline for at least 15 years.
IMHO old-school Google remains the high water mark of generalized information retrieval, with advantages ranging from speed to semi-durable citation.
I strongly suspect there is a cohort thing going on here, many HN users today weren’t involved in technology yet back when Google worked well.
Much like beer for Homer Simpson, AI is the cause of and solution to all of the Internet’s problems.
emtel
I’ve been using Google since about 2000/2001 or so. In fact I worked there for a time, and I still remember Craig Silverstein’s answer to “isn’t search a solved problem?” Which was: “in the future we want Google to be able to take any question and just give you the answer. So clearly we have a long way to go!”
In any case, I do not believe there was ever a time it could answer all of the questions that LLMs can today. If the question had been asked and answered on the web, Google could (and can) find it, but many questions haven’t been asked!
kortilla
Search isn’t the same thing as answering a question
nuancebydefault
Google search works at least as well as 15 years ago, the results are now just more interleaved with ads that are harder to distinguish from what you are really searching for.
That's exactly where LLMs come in, the model inside the weights has more than answers, they can find sense in data.
OptionOfT
I think those are 2 different issues.
Searching for something, and finding it, is different from what ChatGPT / Claude does.
Google (in the good old days) is like the library. You want to search 'how to plant potatoes on Mars'. No results. Well, you split it up, maybe a book on planting potatoes, and a book about missions to Mars that describes soil composition.
Then, when you have those books you start reading, parsing, understanding, making connections, identifying what needs to be done etc.
Maybe, if you're lucky, you find a book or a web page where somebody went through the thought exercise of finding out what would be needed to make it work.
ChatGPT / Claude / ... are different in that they have the information in their corpus, and that the information they present you could actually be something that has never been written down in a book, or published on the web. That's why Google can't find it, but ChatGPT is able to present you with a satisfying answer.
Now whether the answer is correct is a different issue. Do you have the knowledge to verify this?
inglor_cz
OK, so I asked ChatGPT how to plant potatoes on Mars...
=================================================
Planting potatoes on Mars would be a pretty fascinating (and challenging) task! While Mars has conditions that make growing traditional crops difficult—like low temperatures, low atmospheric pressure, and a lack of oxygen—scientists have been experimenting with ways to grow plants in Martian-like conditions. Here’s an overview of the process:
1. Create a Controlled Environment:
Temperature: Mars’ average temperature is about -80°F (-60°C), far too cold for potatoes to grow. You’d need to create a greenhouse-like environment, potentially using materials like glass or transparent plastics, to keep the temperature warmer.
Atmosphere: Mars has only about 1% of Earth's atmospheric pressure, which is too low to support plant life. A sealed greenhouse would be required to provide a breathable atmosphere with adequate pressure and oxygen levels.
Light: Mars has less sunlight due to its distance from the Sun. You would need supplemental artificial light (perhaps LEDs) to mimic Earth’s daylight cycle for the plants.
2. Soil: Mars has soil, but it’s not exactly like Earth’s. It lacks organic material and has toxic elements like perchlorates that would harm plant growth. Scientists would need to either:
Modify Martian soil by adding organic material (like compost) and removing or neutralizing toxic chemicals.
Hydroponics: Grow potatoes without soil, using a nutrient-rich water solution instead.
unification_fan
You can ask it whatever you want but unless you do some pen and paper calculations to prove that whatever GPT says isn't bullshit, you're just writing fanfiction.
dexwiz
What is this supposed to prove? The question isn’t particular novel. There are decades of speculative terraforming discussions on the internet that you can search via Google, especially after the Martian book and movie.
BeetleB
The comparison with search is faulty to begin with. Yes, you can search with an LLM, but that's a side effect of the tool.
While I certainly also have found things via LLMs that I couldn't easily with a search engine, the number of false positives is huge. My heuristic is:
If I ask an LLM something and it's easy to verify via Google because its answer narrows the search space - then I'll use it. Otherwise, Google is still king.
Example: Asking an LLM the health benefits of supplement X is a waste of time. Verifying everything it tells me would be the same amount of work as asking a search engine.
Example: Asking how to solve a given coding problem is great, because it drastically reduces the search space. I only have to look up the particular function/API calls it uses.
Ditto for asking how to achieve a task in the command line - I can quickly verify the arguments are accurate via the man page.
Most of the things I search for do not fall into this category, but in the category of "still need to do the same amount of work as just searching via Google."
bmitc
What are the examples?
I've had several LLM search result summaries contain flat out mistakes and incorrect statements.
emtel
I’ll try to dig some up soon (I’m on my phone now). But of course the output contains errors sometimes. So do search engine results. The important thing for difficult questions is whether the right answer (or something pointing toward it) is available _at all_. Of course this assumes you can verify the answers somehow (usually easy with programming questions), but again, search engines have the same limitation.
vhantz
> But of course the output contains errors sometimes. So do search engine results.
That's not true.
Search engine results are links and (non-AI generated) summaries of existing resources on the web. No search engine returns links to resources it generated as the result of the search query. Those resources can have innacurate information, yes, but the search engine itself does not returns errors.
LLMs output do not contain errors "sometimes". The output of an LLMs is never truthful nor false in itself. In the same way that the next word your keyboard suggests for you to type on a mobile device is never truthful nor false. It's simply the next suggestion based on the context.
These two methods of accessing information very clearly do not have the same limitations. A search engine provide link to specific resources. A LLM generates some approximation of some average of some information.
It's up to intelligent thinking people to decide whether a LLM or a search engine is currently the best way for them to parse through information in search for truth.
emtel
Ok, the first example I found was when I was trying to find a way to write a rust proc macro that recursively processes functions or modules and re-writes arithmetic expressions. The best way to do this, it turns out, is with `VisitMut` or `fold`. I cannot find any results discussing these approaches with google, but ChatGPT (4) suggested it within the first couple refinements of a query.
Another recent example from my history: "can you implement Future twice for a rust struct, with different Output types"
bmitc
There isn't an expectation or claim that search engines answer anything. They just find things or don't find things.
timewizard
I've had several summaries that are just 80% duplications of pages found in the 4th to 5th position in the search results.
It seriously looks like google is deranking actually useful and informative sites and then passing their content through an "LLM" to slightly reorganize it and then pass it off as it's own.
It's a copyright laundering machine put together by advertising companies so you never leave their properties. I genuinely think it's a criminal conspiracy at this point.
vrnvu
Highly recommend "Stories of Your Life and Others".
I describe Ted Chiang as a very human sci-fi author, where humanity comes before technology in his stories. His work is incredibly versatile, and while I expected sci-fi, I'd actually place him closer to fantasy. Perfect for anyone who enjoys short stories with a scientific, social, or philosophical twist.
Another anthology I'd recommend with fresh ideas is Axiomatic by Greg Egan.
m_fayer
While he’s very much unique, the one writer he brings to my mind is Borges, just a bit more humane and steeped in real technology and theory.
justinpombrio
Chiang writes science-fiction, Egan writes physics-fiction, and Borges wrote philosophy-fiction.
Squarex
Ted Chiang and Greg Egan are my absolutely favourite authors. Do you know about other similar writers?
smallerfish
Exurb1a is also worth reading. He's better known for his YouTube video essays (which vary between bleak and profound, usually within the same video), but he has published several books. I got about halfway through Fifth Science before leaving it on a plane (yesterday); I plan to rebuy it so that I can finish it.
Here's one of his stories: https://www.youtube.com/watch?v=sKouPOhh_9I
baruchel
Some stories by Ted Chiang share similarities with those of Borges.
vrnvu
Love Borges.
In the sci-fi space I'd argue that Ursula K. Le Guin is another must read. She was heavily influenced by taoism (and eastern philosophy). When you approach her work with that in mind, it adds a whole new layer of depth to everything.
Squarex
Could you share some tips where to start with him?
ycombinete
I’ve never encountered anything like Egan before. I’ve heard Stanislaw Lem mentioned in conversations about him though. But I can’t vouch for the comparison myself as I’ve never read Lem.
JadeNB
Both are fresh voices and well worth reading, but I don't think Lem comes anywhere near Egan's diamond-hard sci-fi. Egan knows, and does, real math; you can sometimes find him at the n-category Café. My impression is that Lem's beautiful philosophical ideas were not accompanied by comparable math or physics knowledge.
andrei_says_
A bit “harder” sci-fi but incredible world building - Alastair Reynolds.
I recommend his short stories first - Galactic North is a good start. Or Beyond the Aquila Rift.
House of Suns is a good first novel.
bookofjoe
The last time I responded to a similar comment by suggesting asking an AI, I was downvoted to hell. I won't do it again. I will note, though, that the list generated was excellent and provided rewarding information.
Squarex
I've tried but it has provided pretty generic authors I've known and are great, but are not at all that similar to those two. https://gist.github.com/jansuchomel/b3da1f8e2588e13ba9fc0059...
Any ideas for a better prompt?
breakitmakeit
As a fan of both I highly recommend Adrian Tchaikovsky's Children of Time (https://en.wikipedia.org/wiki/Children_of_Time_(novel)) for something deeply imaginative, empathetic, and technically plausible.
ericrallen
I also really enjoyed Chiang’s “Exhalation” anthology.
There are some great short stories in both collections.
satvikpendem
I recommend the story Hell is the Absence of God in the book you mentioned; as someone non-religious, it was quite interesting to see how people generally feel about deities and their awesome power, from this short story [0].
[0] https://static1.squarespace.com/static/50e08e65e4b0c2f497697...
RangerScience
I think of this as “humanist” sci-fi; which has heavy overlap with “golden era” SF.
Other authors I’d put in this category are Gene Roddenberry (TOS and TNG, particularly), Asimov, PKD, Vonnegut and Theodore Sturgeon.
Personally - fantasy stories are “and-then” stories, SF are “what-if”. Humanist sci-fi is then asking “what-if” about very human things, as opposed to technological things, although the two are always related.
However, practically speaking, literature vs sci-fi vs fantasy (vs young adult!) are more marketing cohorts than anything else; what kind of people buy what kind of books?
gcanyon
I'll take this as my chance to recommend Ted Chiang -- he is among the very best short story writers working in science fiction (I say confidently, not having done an extensive survey...). His works are remarkably clever, from Understand, which does a credible job of portraying human superintelligence, to Exhalation, which explores the concept of entropy in a fascinating way. And of course Story of Your Life, on which Arrival was based.
Almost all of his stories are gems, carefully crafted and thoughtful. I just can't recommend him enough.
3abiton
I second Exhalation, it was a great experience, and I couldn't share that excitement after reading as no one in my (offline) circle had read it at the time. Reddit was one of those places where it was brought up at least.
gcanyon
It's a great metaphor, and such a ridiculous path to get there (when the researcher is (mild spoilers) experimenting on himself. So good!
7thaccount
Understand really was my favorite super intelligence story. Isn't the new show severance based off one of his short stories as well? I can't remember. I wasn't a big fan of the pacing of the digients story with the little digital life forms. It was certainly thought provoking though.
gcanyon
It's sooo easy to blow it when illustrating super intelligence. Interestingly, I thought Phenomenon with John Travolta did a pretty reasonable job -- when they asked him to name as many animals as he could in a minute, and he was bored and made it harder (by giving them for each letter of the alphabet?) and when he says to a friend that he should redesign/paint the parking lot outside his store because he could add an extra space and at the same time make the traffic flow smoother.
jrussino
I'll preface this by saying I've only watched season 1 of Severance so far.
I have never heard anyone involved with the show suggest this, but I feel pretty strongly that it's based off or at least inspired by Greg Egan's "Learning To Be Me".
I-M-S
Severence is an original work written by Dan Erickson.
null
OisinMoran
Yeah I’m a massive Chiang fan but I’ve told everyone to skip “The Lifecycle of Software Objects” when recommending the Exhalation collection, though I feel it’s only getting more relevant.
mkoubaa
Why? I found that story fantastic
owlninja
I first heard of him at some point on HN. Tower of Babylon absolutely blew me away. I read all his other story collections and can't recommend them enough.
mkoubaa
I read the Exhalation collection last year and plan to read it once every two or three years for the rest of my life
gcanyon
I have stories on permanent rotation like this.
textlapse
I recommend reading George Saunders - probably the best living short story writer. His way of writing is both out there and grounded at the same time. Not much sci-fi (if anything it’s whimsical) though.
His collection Tenth of December is probably my favorite.
null
dshacker
Ted Chiang is one of my favorite novelists. His way of writing is mentally engaging and FUN. One of my favorite books is his compendium of short stories "Exhalation". My favorite story is the one where you can talk/interact/employ your alternative selves from other universes. Highly recommend.
photochemsyn
> “It’s like imagining that a printer could actually feel pain because it can print bumper stickers with the words ‘Baby don’t hurt me’ on them."
Problem: the human brain has no pain receptors, no nociceptors. It just takes in messages from remote nerves and 'prints a bumper sticker' that tells higher cognitive centers 'you're feeling pain!'. What's the difference?
> "LLMs are like a search engine that rephrases information instead of giving it verbatim or pointing you to the original source."
Problem: How does this differ from human learning? If a human reads a book and tells someone else about it, constructs a summary of the important points and memorable passages, how is that fundamentally different from what LLMs are doing?
The second one really impacts the intellectual property arguments - if training a model on data is fundamentally similar to training a human on data, does 'derivate work' really apply to the creations of the human or of the model?
Barrin92
>What's the difference?
The pain receptors. The human brain doesn't just "have" pain receptors. Your entire body, including your brain, is one system. Your brain isn't piloting your body like a mech. This brain body dualism is a misconception of how biological organisms work. You are your pain receptors just like you are your brain, and removing any part would alter your perception of the world.
>How does this differ from human learning?
It differs from human beings in every respect. Humans don't do linear algebra in their head, biochemical systems are much too slow for that. Humans don't inhabit some static model of the world learned at some fixed point t, you're a living being. Your brain wasn't trained four months ago and was done at that point. Humans learn with a fraction of the information and through self play, they don't decohere, and so on.
photochemsyn
If you were technologically advanced to the point of being able to float a human brain in a vat, and connect up all the nerves going into that brain to synthetic sensors with the appropriate signalling outputs, then you could construct a synthetic reality for that brain that it would not be able to distinguish from its previous brain-in-a-body experiences.
As far as learning, human learning is certainly much slower than machine learning but it's not really clear at a biochemical-molecular level that they're entirely different, eg the formation of memories and so on, considering a wide range of alternate hypothesis before selecting one, etc.
Barrin92
>you could construct a synthetic reality for that brain that it would not be able to distinguish from its previous brain-in-a-body experiences.
No. I'd recommend reading Dennett's Consciousness Explained for a longer treatment of this, but if you want to have an experience just like you, you need a body and an environment just like you. Sure it could be synthetic in the sense of it being artificially constructed, you could have artificial limbs, but it can be no different from the one you have, it cannot be a vat. There is no "Cartesian Theater" in the brain, your experience is already distributed throughout both your brain and body. Your experience isn't something being "fed into you" (the brain) from the outside, the entire perception emerges in the first place through being embodied in the world. The concept of the thought experiment itself would not even make sense without implicitly assuming an embodied reality beforehand setting it up.
Just like there is no philosophical zombie that's somehow materially identical but without sentience, the reverse doesn't exist either. There is no disembodied computer with the experiences of an organic being because they function in entirely different ways.
paulryanrogers
Humans are far more self contained than most LLMs which depend upon a lot of electricity and possibly an array of loosely connected components. LLMs also don't really have the signal bandwidth of human nervous systems yet. They're capacity to interact with the physical world is also seriously limited for now.
rednafi
Ted Chiang is a master of analogies. It’s absolutely delightful to read his work and wrestle with the philosophical questions he explores. I devour almost everything he puts out, and they give me a much-needed escape from my world of bits and registers.
“LLMs are a blurry JPEG of the web” has stuck with me since the piece was published in the early days of ChatGPT. Another good one is his piece on why AI can’t make art.
While I heavily use AI both for work and in my day-to-day life, I still see it as a tool for massive wealth accumulation for a certain group, and it seems like Ted Chiang thinks along the same lines:
> But why, for example, do large corporations behave so much worse than most of the people who work for them? I think most of the people who work for large corporations are, to varying degrees, unhappy with the effect those corporations have on the world. Why is that? And could that be fixed by solving a math problem? I don’t think so.
> But any attempt to encourage people to treat AI systems with respect should be understood as an attempt to make people defer to corporate interests. It might have value to corporations, but there is no value for you.
> My stance on this has probably shifted in a negative direction over time, primarily because of my growing awareness of how often technology is used for wealth accumulation. I don’t think capitalism will solve the problems that capitalism creates, so I’d be much more optimistic about technological development if we could prevent it from making a few people extremely rich.
est
> master of analogies
analogy, in other words, embeddings?
the_af
No, analogies and embeddings are not exactly the same. Analogies in language are not a math function.
As Ted Chiang comments on the article, this kind of reasoning ("the brain is like $CURRENT_TECH") is flawed.
est
well, embedding comes with certain loss as well.
If you believe our brains use "language" to think, then I would assume analogies play an important part in reasoning.
leedrake5
> as in your 1991 story “Division by Zero,” or a world where we raise robots as children
This is vastly more preferable than our current approach of raising children as robots.
bmitc
Humans love the pursuit of technology for technology's sake.
Quotes by Jacques Ellul:
----
> Technique has taken over the whole of civilization. Death, procreation, birth all submit to technical efficiency and systemization.
----
> Technique has penetrated the deepest recesses of the human being. The machine tends not only to create a new human environment, but also to modify man's very essence. The milieu in which he lives is no longer his. He must adapt himself, as though the world were new, to a universe for which he was not created. He was made to go six kilometers an hour, and he goes a thousand. He was made to eat when he was hungry and to sleep when he was sleepy; instead, he obeys a clock. He was made to have contact with living things, and he lives in a world of stone. He was created with a certain essential unity, and he is fragmented by all the forces of the modern world.
throwaway2037
I didn't know Ted Chiang before seeing this post. If anyone needs to know about him, he has a separate Wiki page: https://en.wikipedia.org/wiki/Ted_Chiang
> Ted Chiang is an American science fiction writer. His work has won four Nebula awards, four Hugo awards, the John W. Campbell Award for Best New Writer, and six Locus awards. Chiang is also a frequent nonfiction contributor to the New Yorker, most recently on topics related to computer technology, such as artificial intelligence.vicentwu
"As an analogy, imagine that you could put your dog or cat into hibernate mode whenever you left on a trip. Your dog or cat might not notice, but even if they did, they might not mind. Now imagine that you could put your child into hibernate mode whenever you were too busy to spend time with them. Your child would absolutely notice, and even if you told them it was for their own good, they would make certain inferences about how much you valued them. That’s the situation the human characters in the story find themselves in." Fascinating.
Chiang makes some insightful points, e.g. about what we mean by magic.
Then I come to
> [LLMs] can get better at reproducing patterns found online, but they don’t become capable of actual reasoning; it seems that the problem is fundamental to their architecture.
and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here? If an AI proves a theorem is it only a simulated proof?