Minecraft with object impermanence
42 comments
·January 19, 2025skykooler
kqr
> over five years
I think there is a generation of Minecraft players that participated in the alpha, beta, and early 1.0 releases and then grew out of it. For these people, "in the game for over five years" is basically the same as "new".
(I belong to that group. I recently started introducing my four-year old to Minecraft and there is so much stuff that was not there when I played!)
bakugo
Can confirm.
Whenever I'm reminded of the existence of things like Villagers, or the End, my brain still goes "oh yeah, that new thing they added" before reality catches up and I realize just how long those things have been in the game.
Part of it is because 80%+ of my playtime was before they were added, but I think another part of it is that those things have never really quite fit into the game in my mind.
imtringued
The End is akin to an inside joke or maybe a pun. You cannot finish a sandbox game, so Notch deliberately added a dimension that you have to go out of your way for, that is difficult to find, unless someone tells you or you read about it. The End dimension isn't meant to fit in the game. It's meant to subvert the idea of a sandbox game.
You can say "I've reached the end of Minecraft" and it is true by the letter of the words, but untrue by the spirit of the words. Beating the Ender Dragon does not mean you finished all the content in Minecraft, since there is content even beyond the Ender Dragon.
Akronymus
I stopped playing anything beyond 1.7.10.
I still play a lot of 1.7.10 though, because of gtnh. Loving that modpack quite a bit.
littlestymaar
This exactly. I wouldn't be surprised if most HN readers had played a bit of Minecraft 13 to 15 years ago and still represent Minecraft like that even after all that time. I'm definitely one of those BTW.
TeMPOraL
Same here. Played a lot of it in a brief moment some 13 years ago, then never touched it again. Villagers and stuff is still strictly in the "new" category for me.
sedatk
That's pretty much how my dreams work. For example, I can read in my dreams (like signs, books), but it's almost always one word at a time, and the rest of the text becomes impossible to focus while I'm reading that word, no matter how hard I try. It's like the "permanence module" is disabled and my brain falls back to a primitive generative AI for the next frame. And, I can't reason about the logic of my experience, the lack of causality, or persistence of objects at the time. Everything feels "as they should be" despite how awkward things get. I guess that's why nightmares are so convincing or hard to get out of, because I can't reason about them. The whole article feels like describing a typical dream to me.
HappMacDonald
My experience with nightmares has largely been that a part of my subconscious is trying to conduct the dream in an "exciting" direction, to engender thrill on a par with a carnival ride .. but then it slips or gets out of hand and turns into a spiral of self-fulfilling terror instead. I feel like ruminating on that process some during waking time has helped my dream producing instincts to instead steer away from unnecessary sources of fear often before I even perceive (during the dream) that things might have been about to get scary (though I'm often able to assess that pattern after the fact).
EZ-E
> I can read in my dreams (like signs, books), but it's almost always one word at a time, and the rest of the text becomes impossible to focus while I'm reading that word, no matter how hard I try. It's like the "permanence module" is disabled
Techniques for lucid dreaming rely on this fact for you to be able to "detect" that you are in a dream. There is one where you need to look at time, look away and look again. The time will change or be in an impossible format if you're dreaming. Then if you're aware you're dreaming you can start to "control" the dream.
rogual
Recently I found myself looking at an old alarm clock with a seven-segment display, but the segments were lit up arbitrarily, making an unreadable pattern instead of a time. But then I remembered, oh yeah, that's right, I do have a broken old alarm clock that does that, it's probably just that, I'm not dreaming. And I fell back into the dream.
The funny thing is, I really do have an old alarm clock that does that. It's the most useless alarm clock ever, because not only can it not tell you the time, it can't even tell you if you're awake.
westmeal
I've found your brain is extremely quick to jump to conclusions in a dream. It's surreal sometimes where random connections like your alarm clock suddenly come out of nowhere and draw you back in.
lucidlucia
Funny enough, I never really needed these tricks. Or I guess, I do have one? It's when I realize the events are scripted. I don't know why, but many of my dreams are perfectly sensical and linear, and sometimes I notice and hijack them. What I can do with them tho, depends on how close I am to waking up.
Saddest one was when I realized I was walking my dead dog. She was actually exhibiting behaviors I know she had been used to have, but not ones I had consciously remembered much when thinking about her after her death. I lingered on that one a bit. Still, dreams are dreams, and there was some weirdness, and I knew them for what they were. But that time, I did not fight the script.
kqr
> It's like the "permanance module" is disabled and my brain falls back to a primitive generative AI for the next frame.
What if there is no such module and whenever you believe you have experienced permanence it is because your brain confabulated it from what it actually observed?
sedatk
I'd still call that permanence module :) AFAIK, that the way our vision works is that our eyes only see a narrowly focused region in detail and the rest of our periphery is generated by our brain (based on what we saw there before). There are optical illusions based on that.
xeonmc
"Attention", perhaps?
rogual
There's a bit in Ender's Game where he plays this computer game that has a dreamlike quality to it. It seemed it could simulate a response to any player action, and continue the story along that path. I think in the book it was a tool for introspection, like dreams can be? I can't really remember.
Anyway, as a kid I thought, silly author, that's not how computer games work. But it turns out Orson Scott Card is smarter than me. Give this kind of thing a few years and we'll have it.
lou1306
> Things became duplicated in Tlön; they also tend to become effaced and lose their details when they are forgotten. A classic example is the doorway which survived so long as it was visited by a beggar and disappeared at his death. At times some birds, a horse, have saved the ruins of an amphitheater.
JL Borges, Tlön, Uqbar, Orbis Tertius
fdb
Janelle Shane’s work has been really inspiring. Her talk at Strange Loop from a couple of years ago — right before DALL-E, before ChatGPT — was a turning point, and very funny:
rollcat
That's the difference between a deterministic and a stochastic system. Using one where you need the other leads to exactly these kinds of results. Recent trend in LLMs is to gravitate towards more determinism (like fact checking), I'm afraid in the end we will come full circle and end up with a fancy frontend to a database.
torginus
The AI dungeon thing is a salient point - GPT2 (which it started out with) could barely make a coherent paragraph, and GPT3 could write about half a page of text that more or less made sense.
With modern LLMs, they still get occasionally tripped up, but you could go for pages without a minor detail not making sense.
Something similar might happen with these game models, given enough time.
TeMPOraL
AI Dungeon was the text equivalent of the Minecraft thing in TFA. I still remember the distinct feeling I had after getting immersed in the interactive story experience for an hour - for the rest of the day, I felt like I woke up from an intense fever dream.
folli
So the AI is trained from the previous frame and the input and tries to predict the next frame, correct?
How could you achieve object permanence this way? Will it 'automatically' appear given more training data or more hidden layers? How is this handled in other approaches?
kvdveer
If training is indeed done on frame + input, any information that isn't in either of those data sources is simply not there.
To achieve object permanence, there needs to be some persistent off-screen data from frame to frame. There's a way to achieve this: train on frame+input+woldstate -> frame+woldstate
383toast
the same way LLMs are trained to predict next tokens from prev tokens, just longer context = better memory = object permanence
echoangle
But then you can’t just give the previous frame, with the LLM analogy you would have to give the last few thousand frames (that’s the context window, right?). If you only give the previous frame, that’s like having an LLM that only gets the single previous token and has to predict the next one.
int_19h
Indeed. Although more recently they figured out a way to feed the hidden state as the new input, which basically allows the model to "continue thinking" in vectors without round-tripping it via words (or pixels).
Presumably if you were to take that and build a large enough NN to accommodate all the necessary state it needs to carry and all the rules it needs to be able to execute, then after training it on enough game input you'd have a proper world simulation. Of course, as the article rightly notes, then you have just successfully reimplemented Minecraft in a way that is orders of magnitude more computationally expensive...
_flux
Perhaps the trick used by text-based LLMs could be used: when the context window starts filling up, the LLM is asked to summarize the existing data in the context, thus compressing it (lossily..) into smaller space.
Sharlin
More previous tokens in this case would mean more previous frames. But there's really no reason to just stick to rendered pixels as input (except for novelty's sake) because we could train directly on snapshots of full game state.
omcnoe
Interesting blog post from the creators with more details about the model: https://oasis-model.github.io
Interactive demo: https://oasis.decart.ai/
Matthyze
I really adore this. AI art is at its best when it's strange and uncanny. Another example (watch till the end):
https://www.youtube.com/watch?v=XqLm0qbLJqY
edit: I'll add this great take too:
cubefox
Apparently Oasis Minecraft is what's called a Markov process: The next state only depends on the current state, but not on the past states. To "fix" this, Oasis Minecraft would need a memory of the past frames.
_def
I kind of want to see this with Pokemon gen 1 games now.
This was a fun read. But some of the things attributed to a glitchy AI are just normal Minecraft stuff. For example: "One time I was swimming across a lake and noticed that the reflections at the water's edge were looking weirdly spiky." That's kelp, it looks like that in the game because the chunks beyond haven't loaded so it stands out from the sky. Also, the "natural blocks of rare glowstone" are honeycomb blocks, which she thinks might be new "cheese" blocks but in fact have been in the game for over five years.