Qwen2.5-1M: Deploy your own Qwen with context length up to 1M tokens

I concur. In my work (analysing news show transcripts and descriptions), I work with about 250k input tokens max. Tasks include:

- Summarize topics (with references to shows) - Find quotes specific to a topic (again with references)

Anything above 32k tokens fails to have acceptable recall, across GPT-4o, Sonnet, and Google's Gemini Flash 1.5 and 2.0.

I suppose it kind of makes sense, given how large context windows are implemented via things like sparse attention etc.

My hypothesis is code completion is not a text completion problem. More of a graph completion one.

So we may have got to a local maximum regarding code helpers with LLMs and we'll have to wait for some breakthrough in the AI field before we get something better.

Aider is great, but you need specific formats from the llm. That might be where the challenge is.

I've used the giant context in Gemini to dump a code base and say: describe the major data structures and data flows.

Things like that, overview documents, work great. It's amazing for orienting in an unfamiliar codebase.

I learned this very explicitly recently. I've had some success with project and branch prompts - feeding a bunch of context into the beginning of each dialog.

In one dialog, some 30k tokens later, Claude requested the contents of package.json... which was in the context window already - the whole file!

The strange thing was that after I said so, without re-inserting, Claude successfully read it from context to fill the gap in what it was trying to do.

It's as if a synopsis of what exists in-context delivered with each message would help. But that feels weird!

Any idea why this happens?

It’s not just the quantity of tokens in context that matters, but the coherence of the concepts in the context.

Many conflicting ideas are harder for models to follow than one large unified idea.

The behaviour you described is what happens when you have small context windows. Perhaps you're feeding the models with more tokens than you think you are. I have enjoyed loading large codebases into AI Studio and getting very satisfying and accurate answers because the models have 1M to 2M token context windows.

Overall accuracy degradation on longer contexts is just one major issue. Another is that lost-in-the-middle problem starts being much worse on longer contexts, so when it significantly exceeds the length of model's training examples, the tokens in the middle might as well not exist.

This is a harsh foot-gun that seems to harm many ollama users.

That 2k default is extremely low, and ollama *silently* discards the leading context. So users have no idea that most of their data hasn’t been provided to the model.

I’ve had to add docs [0] to aider about this, and aider overrides the default to at least 8k tokens. I’d like to do more, but unilaterally raising the context window size has performance implications for users.

Edit: Ok, aider now gives ollama users a clear warning when their chat context exceeds their ollama context window [1].

[0] https://aider.chat/docs/llms/ollama.html#setting-the-context...

[1] https://github.com/Aider-AI/aider/blob/main/aider/coders/bas...

Huh! I had incorrectly assumed that was for output, not input. Thanks!

YES that was it:

  files-to-prompt \
    ~/Dropbox/Development/llm \
    -e py -c | \
  llm -m q1m 'describe this codebase in detail' \
   -o num_ctx 80000

Ollama is a "easymode" LLM runtime and as such has all the problems that every easymode thing has. It will assume things and the moment you want to do anything interesting those assumptions will shoot you in the foot, though I've found ollama plays so fast and loose even first party things that "should just work" do not. For example if you run R1 (at least as of 2 days ago when i tried this) using the default `ollama run deepseek-r1:7b` you will get different context size, top_p and temperature vs what Deepseek recommends in their release post.

Yup, and this parameter is supported by the plugin he's using:

https://github.com/taketwo/llm-ollama/blob/4ccd5181c099af963...

MLX does not support dual chunk attention[1] that these models use for long contexts, yet.

[1]: https://arxiv.org/abs/2402.17463

I'm not sure how SOTA it is but the sentence about connecting DRAM differently reminded me of Cerebras' scalable MemoryX and its "weight streaming" architecture to their custom ASIC. You may find it interesting.

[1]: https://cerebras.ai/press-release/cerebras-systems-announces...

[2]: https://cerebras.ai/chip/announcing-the-cerebras-architectur...

The AI hardware race is still going strong, but with so many rapid changes to the fundamental architectures, it doesn't make sense to bet everything on specialized hardware just yet.. It's happening, but it's expensive and slow.

There's just not enough capacity to build memory fast enough right now. Everyone needs the biggest and fastest modules they can get, since it directly impacts the performance of the models.

There's still a lot of happening to improve memory, like the latest Titans paper: https://arxiv.org/abs/2501.00663

So I think until a breakthrough happens or the fabs catch up, it'll be this painful race to build more datacenters.

> Because racks of H100s are not sustainable.

Huh? Racks of H100s are the most sustainable thing we can have for LLMs for now.

No, this is not the first local model with a context length of greater than 128k, there have been such models, for example the following

https://huggingface.co/ai21labs/AI21-Jamba-1.5-Mini 256k https://huggingface.co/THUDM/glm-4-9b-chat-1m 1M

and many other's that supposedly extended traditional models via finetune/rope scaling

Yes. It requires a lot of ram, and even on a M4 with a lot of ram, if you give it 1mio tokens the prompt processing alone (that is, before you get the first response token) will probably take ~30min or more. However I'm looking forward to check if indeed I can give it a whole codebase and ask questions about it.

I gave it a 446433 token input, then it calculated for ~4 hours, and gave me a reasonable response. The content was a Rust / Typescript codebase where Typescript is the frontend and Rust is the backend. I asked it which backend apis are currently not used by the frontend. I haven't checked yet, but the answer looked correct.

Running this on a M4 max

I bought an M4 Max with 128g of ram just for these use cases. Currently downloading the 7b

I'm missing what `files-to-prompt` does. I have an M3 Max and can take a stab at it, although I'm currently fussing with a few quantized -r1 models...

That's a different model - the 2.5 Turbo one. Today's release is something different.

You can't on modest hardware, VRAM size is a function of model size, KV cache that depends on context length and the quant size of the model and K/V. 16gb isn't much really. You need more vram, the best way for most folks is to buy a macbook with unified memory. You can get a 128gb mac, but it's not cheap. If you are handy and resourceful you can build a GPU cluster.

You need a model that has specifically been extended for larger context windows. For Llama-3 there's Llama3-gradient with up to 1M tokens. You can find it at ollama.com

Repeatedly ask it for more providing the previous output as context. (Back to context length as a limitation)

Isn't that the same? limit-wise.

Now you just need to convince it to output that much :)

So context size actually helps with this, relative to how LLMs are actually deployed as applications. For example, if you look at how the “continue” option in the DeepSeek web app works for code gen, what they’re likely doing is reinserting the prior messages (in some form) to a new one to prompt further completion. The more context size a model has and can manage successfully, the better it will likely be able at generating longer code blocks.

These things already produce embarrassing output. If you make it longer it's just going to get worse.

Chain of agents seems to be a promising approach for splitting up tasks into smaller parts and then synthesising the results[1]

[1] https://research.google/blog/chain-of-agents-large-language-...