Google Titans architecture, helping AI have long-term memory

DeepSeek and other Chinese companies. Not only do they publish research, they also put their resources where their mouth (research) is. They actually use it and prove it through their open models.

Most research coming out of big US labs is counter indicative of practical performance. If it worked (too) well in practice, it wouldn't have been published.

Some examples from DeepSeek:

https://arxiv.org/abs/2405.04434

https://arxiv.org/abs/2502.11089

Well it's cool that they released a paper, but at this point it's been 11 months and you can't download a Titans-architecture model code or weights anywhere. That would put a lot of companies up ahead of them (Meta's Llama, Qwen, DeepSeek). Closest you can get is an unofficial implementation of the paper https://github.com/lucidrains/titans-pytorch

Bytedance is publishing pretty aggressively.

Recently, my favorite from them was lumine: https://arxiv.org/abs/2511.08892

Here's their official page: https://seed.bytedance.com/en/research

Meta is also being pretty open with their stuff. And recently most of the Chinese competition.

> Is there any other company that's openly publishing their research on AI at this level? Google should get a lot of credit for this.

80% of the ecosystem is built on top of companies, groups and individuals publishing their research openly, not sure why Google would get more credit for this than others...

It was not always like this. Google was very secretive in the early days. We did not start to see things until the GFS, BigTable and Borg (or Chubby) papers in 2006 timeframe.

Every Google publication goes through multiple review. If anyone thinks the publication is a competitor risk it gets squashed.

It's very likely no one is using this architecture at Google for any production work loads. There are a lot of student researchers doing fun proof of concept papers, they're allowed to publish because it's good PR and it's good for their careers.

The author is listed as a "student researcher", which might include a clause that students can publish their results.

Here is a bit more information about this program: https://www.google.com/about/careers/applications/jobs/resul...

Hey it was my turn to post this quote today!

This is an oversimplification of what Titans does. The model performs nested learned, where the model learns during inference, and during training the model weights learn _how and what_ to learn during inference. If the input contains junk of irrelevant information, the model most likely learned during training to assign low surprise query and key embeddings to those tokens, because learning those junk tokens would have hurt the overall ability of the model to predict subsequent next tokens (and thus, it would have had increased the training loss).

I’m guessing that this is the first thing they thought of and the problem only exists in the superficial gloss you’re responding to?

In what world can you not always break the response of an AI by feeding it a bunch of random junk?

The is the start of what I always thought an AI should have - a limbic system. Humans don't store memory based on novelty, they store it based on emotional content. This is where I was afraid of the tiger, this is where I smelled delicious food, this was what it felt like when I was victorious in the hunt.

AI needs an internal emotional state because that's what drives attention and memory. AI needs to want something.

This is no different from what happens to humans if they're locked into cult programming situations, they'll start believing and regurgitating all kinds of nonsense if their information stream is tightly curated,

Practically, for use with a codebase development effort, if the model remembers the original design decisions, the discussions about costs and benefits, then can remember all that much later in the process, it's going to start getting really good at thinking about what the next step is, or even to make decisions about when a major refactor is neede, etc.

Sharing it in your blog over a period of months or years is how you become a known figure eventually.

A lot of LLM/AI writing these days can feel lost in the weeds – the specifics of very detailed techniques are interesting undoubtedly, but writing that steps back and looks at the big picture, informed by those details, could be very useful for people who want to think about where this all may be going.

Are you curious to see whether a blog post shared here might gain any traction and perhaps some valuable feedback?

Are there any pretrained models with this architecture yet or is it all still completely theoretical beyond Google's unverifiable claims? They published the original Titans paper last year and nobody seems to have built on the idea.

That's not what they are talking about here. This is just a description of what goes on with a transformer and the context window

I don't think that's a great analogy.

LoRAs tend to be adapters bolted onto to systems by people other than the system designers, and they are low rank factorizations.

There is nothing low rank or adapter here.

Kind-of. You could theoretically use LoRA for this, in fact, but it probably wouldn't have enough capacity to make it a proper substitute of the attention mechanism. Instead a full MLP is trained as input chunks get processed.