LLM Daydreaming

From The Metamorphosis of Prime Intellect (1994):

> Among Prime Intellect's four thousand six hundred and twelve interlocking programs was one Lawrence called the RANDOM_IMAGINATION_ENGINE. Its sole purpose was to prowl for new associations that might fit somewhere in an empty area of the GAT. Most of these were rejected because they were useless, unworkable, had a low priority, or just didn't make sense. But now the RANDOM_IMAGINATION_ENGINE made a critical connection, one which Lawrence had been expecting it to make [...]

> Deep within one of the billions of copies of Prime Intellect, one copy of the Random_Imagination_Engine connected two thoughts and found the result good. That thought found its way to conscious awareness, and because the thought was so good it was passed through a network of Prime Intellects, copy after copy, until it reached the copy which had arbitrarily been assigned the duty of making major decisions -- the copy which reported directly to Lawrence. [...]

> "I've had an idea for rearranging my software, and I'd like to know what you think."

> At that Lawrence felt his blood run cold. He hardly understood how things were working as it was; the last thing he needed was more changes. "Yes?"

I’m not sure we can accept the premise that LLMs haven’t made any breakthroughs. What if people aren’t giving the LLM credit when they get a breakthrough from it?

First time I got good code out of a model, I told my friends and coworkers about it. Not anymore. The way I see it, the model is a service I (or my employer) pays for. Everyone knows it’s a tool that I can use, and nobody expects me to apportion credit for whether specific ideas came from the model or me. I tell people I code with LLMs, but I don’t commit a comment saying “wow, this clever bit came from the model!”

If people are getting actual bombshell breakthroughs from LLMs, maybe they are rationally deciding to use those ideas without mentioning the LLM came up with it first.

Anyway, I still think Gwern’s suggestion of a generic idea-lab trying to churn out insights is neat. Given the resources needed to fund such an effort, I could imagine that a trading shop would be a possible place to develop such a system. Instead of looking for insights generally, you’d be looking for profitable trades. Also, I think you’d do a lot better if you have relevant experts to evaluate the promising ideas, which means that more focused efforts would be more manageable. Not comparing everything to everything, but comparing everything to stuff in the expert’s domain.

If a system like that already exists at Jane Street or something, I doubt they are going to tell us about it.

I also hope we have something like this. But sadly, this is not going to work. The reason is this line from the article, which is so much harder that it looks:

> and a critic model filters the results for genuinely valuable ideas.

In fact, people have tryied this idea. And if you use a LLM or anything similar as the critic, the performance of the model actually degrades in this process. As the LLM tries too hard to satisfy the critic, and the critic itself is far from a good reasoner.

So the reason that we don't hear too much about this idea is not that nobody tried it. But that they tried, and it didn't work, and people are reluctant to publish about something which does not work.

> You are a creative synthesizer. Your task is to find deep, non-obvious, and potentially groundbreaking connections between the two following concepts. Do not state the obvious. Generate a hypothesis, a novel analogy, a potential research question, or a creative synthesis. Be speculative but ground your reasoning.

> Concept 1: {Chunk A} > Concept 2: {Chunk B}

In addition to the other criticisms mentioned by posters ITT, a problem I see is: What concepts do you feed it?

Obviously there's a problem with GIGO. If you don't pick the right concepts to begin with, you're not going to get a meaningful result. But, beyond that, human discovery (in mechanical engineering, at least,) tends to be massively interdisciplinary and serendipitous, so that many concepts are often involved, and many of those are necessarily non-obvious.

I guess you could come up with a biomimetics bot, but, besides that, I'm not so sure how well this concept would work as laid out above.

There's another issue in that LLMs tend to be extremely gullible, and swallow the scientific literature and University press releases verbatim and uncritically.

I have not yet seen AI doing a critical evaluation of data sources. AI willcontradict primary sources if the contradiction is more prevalent in the training data.

Something about the whole approach is bugged.

My pet peeve: "Unix System Resources" as explanation for the /usr directory is a term that did not exist until the turn of the millenium (rumor is that a c't journalist made it up in 1999), but AI will retcon it into the FHS (5 years earlier) or into Ritchie/Thompson/Kernigham (27 years earlier).

How do you critique novelty?

The models are currently trained on a static set of human “knowledge” — even if they “know” what novelty is, they aren’t necessarily incentivized to identify it.

In my experience, LLMs currently struggle with new ideas, doubly true for the reasoning models with search.

What makes novelty difficult, is that the ideas should be nonobvious (see: the patent system). For example, hallucinating a simpler API spec may be “novel” for a single convoluted codebase, but it isn’t novel in the scope of humanity’s information bubble.

I’m curious if we’ll have to train future models on novelty deltas from our own history, essentially creating synthetic time capsules, or if we’ll just have enough human novelty between training runs over the next few years for the model to develop an internal fitness function for future novelty identification.

My best guess? This may just come for free in a yet-to-be-discovered continually evolving model architecture.

In either case, a single discovery by a single model still needs consensus.

Peer review?

AlphaEvolve and similar systems based on map-elites + DL/LLM + RL appears to be one of the promising paths.

Setting up the map-elites dimensions may still be problem-specific but this could be learnt unsupervisedly, at least partially.

The way I see LLMs is as a search-spqce within tokens that manipulate broad concepts within a complex and not so smooth manifold. These concepts can be refined within other spaces (pixel -space, physical spaces, ...)

I’m once again begging people to read David Gelernter’s 1994 book “The Muse in the Machine”. I’m surprised to see no mention of it in Gwern’s post, it’s the exact book he should be reaching for on this topic.

In examining the possibility of genuinely creative computing, Gelernter discovers and defends a model of cognition that explains so much about the human experience of creativity, including daydreaming, dreaming, everyday “aha” moments, and the evolution of human approaches to spirituality.

https://uranos.ch/research/references/Gelernter_1994/Muse%20...

It's an interesting premise, but how many people

- are capable of evaluating the LLM's output to the degree that they can identify truly unique insights

- are prompting the LLM in such a way that it could produce truly unique insights

I've prompted an LLM upwards of 1,000 times in the last month, but I doubt more than 10 of my prompts were sophisticated enough to even allow for a unique insight. (I spend a lot of time prompting it to improve React code.) And of those 10 prompts, even if all of the outputs were unique, I don't think I could have identified a single one.

I very much do like the idea of the day-dreaming loop, though! I actually feel like I've had the exact same idea at some point (ironic) - that a lot of great insight is really just combining two ideas that no one has ever thought to combine before.

Regardless of accusations of anthropomorphizing, continual thinking seems to be a precursor to any sense of agency, simply because agency requires something to be running.

Eventually LLM output degrades when most of the context is its own output. So should there also be an input stream of experience? The proverbial "staring out the window", fed into the model to keep it grounded and give hooks to go off?

I'd be happy to spend my Claude Max tokens during the night so it can "ultrathink" some Pareto improvements to my projects. So far, I've mostly seen lateral moves that rewrites code rather than rearchitecture/design the project.

The question is: How do we get LLMs to have "Eureka!" moments, on their own, when their minds are "at rest," so to speak?

The OP's proposed solution is a constant "daydreaming loop" in which an LLM is does the following on its own, "unconsciously," as a background task, without human intervention:

1) The LLM retrieves random facts.

2) The LLM "thinks" (runs a chain-of-thought) on those retrieved facts to see if they are any interesting connections between them.

3) If the LLM finds interesting connections, it promotes them to "consciousness" (a permanent store) and possibly adds them to a dataset used for ongoing incremental training.

It could work.