Gemma 3n preview: Mobile-first AI

Okay from some first tries with story writing, gemma-3n-E4B-it seems to perform between plain Gemma 3 4B and 12B. It definitely retains the strong instruction following which is good.

Hint: You have to set the Max tokens to 32000 for longer conversations. The slider makes it look like it's limited to 1024, just enter it manually.

I assume that "pretty fast" depends on the phone. My old Pixel 4a ran Gemma-3n-E2B-it-int4 without problems. Still, it took over 10 minutes to finish answering "What can you see?" when given an image from my recent photos.

Final stats:

15.9 seconds to first token

16.4 tokens/second prefill speed

0.33 tokens/second decode speed

662 seconds to complete the answer

Why are we still launching models without simple working python example code (or llama.cpp support)?

And the libraries to embed Gemma-series in your iOS/Android app: https://ai.google.dev/edge/litert

Or, run them on a microcontroller! https://github.com/tensorflow/tflite-micro

Thanks for this guide it's great.

Okay perhaps my phones not great and perhaps this isn't optimized/pruned for phone use but it's unusably slow. The answers are solid from my brief test.

I wouldn't exactly say phone use, unless you have no internet and you don't mind a bit of a wait.

Really impressive, regardless.

On Pixel 8a, I asked Gemma 3n to play 20 questions with me. It says it has an object in mind for me to guess then it asks me a question about it. And several attempts to clarify who is supposed to ask questions have gone in circles.

Suggest giving it no networking permissions (if indeed this is about on-device AI).

It's not a 4B parameter model. The E4B variant is 7B parameters with 4B loaded into memory when using per-layer embedding cached to fast storage, and without vision or audio support.

That seems way too good to be true.

What's the catch?

Imagine a model smarter than most humans that fits on your phone.

edit: I seem to be the only one excited by the possibilities of such small yet powerful models. This is an iPhone moment: a computer that fits in your pocket, except this time it's smart.

Hmm, the Aider polyglot benchmark has been removed from the huggingface readme.

Also:

> These models were evaluated at full precision (float32)

For 4B effective parameters that's 16 GB ram.

[flagged]

It's a matter of time before we get a limited activation model for mobile - the main constraint is the raw model size, more than the memory usage. A 4B-A1B should be considerably faster on mobile though, for an equivalent size model (~4Gb).

That sounds like a problem iOS will eventually deal with, as many apps are going to want this technology, and since Apple distributes apps - they aren't interested in the average app being 10x larger when they could solve the problem easily.

Though, I won't be surprised if they try to force devs to use their models for "privacy" (and not monopolistic reasons, of course).

Windows is adding an OS-level LLM (Copilot), Chrome is adding a browser-level LLM (Gemini), it seems like Android is gearing up to add an OS-level LLM (Gemmax), and there are rumors the next game consoles might also have an OS-level LLM. It feels inevitable that we'll eventually get some local endpoints that let applications take advantage of on-device generations without bundling their own LLM -- hopefully.

It seems they are: https://developer.chrome.com/docs/ai/built-in

https://ai.google.dev/gemma/docs/gemma-3n#parameters

> Gemma 3n models are listed with parameter counts, such as E2B and E4B, that are lower than the total number of parameters contained in the models. The E prefix indicates these models can operate with a reduced set of Effective parameters. This reduced parameter operation can be achieved using the flexible parameter technology built into Gemma 3n models to help them run efficiently on lower resource devices.

> The parameters in Gemma 3n models are divided into 4 main groups: text, visual, audio, and per-layer embedding (PLE) parameters. With standard execution of the E2B model, over 5 billion parameters are loaded when executing the model. However, using parameter skipping and PLE caching techniques, this model can be operated with an effective memory load of just under 2 billion (1.91B) parameters, as illustrated in Figure 1.

You can read this for a comprehensive deep dive. https://arxiv.org/pdf/2502.01637

At a very high level, instead of having embeddings at the input layers, this method keeps the embeddings at the layer level. That is every transformer layer would have its own set of learnable embedding vectors that are used to modify the processed hidden states flowing through the network. Mostly, the embeddings are precomputed and stored separately. They are queried at inference time and has very low latency, so you can get comparable performance with half the RAM. (i am not exactly sure how 3n is doing it, but talking it in a general sense).

From the article it appears to be something they invented:

> Gemma 3n leverages a Google DeepMind innovation called Per-Layer Embeddings (PLE) that delivers a significant reduction in RAM usage.

Like you I’m also interested in the architectural details. We can speculate but we’ll probably need to wait for some sort of paper to get the details.

I think that it's a poorly named reference to this paper [1] that they mention in the blogpost. If I had to give it another more descriptive name, I would probably name it "Per-Layer Embedding Dimensionality"

[1] https://arxiv.org/pdf/2310.07707

Per layer LoRA adapters, perhaps? - same as Apple is using for on-device AI.

Took a picture of a bag of chips and it said it was seasoning... I think, for the size, it's OK - but not really there. I'm not sure how they got ELO anywhere near Claude or Gemini... those models are leagues ahead in terms of one-shot accuracy.

Tested it on my Android phone with Google Edge Gallery. No sign up required although a hugging face login is required to download the models in order to import them into the app.

Same here.

I've also seemingly hit a rate limit on Gemini Pro 2.5 (on an account not subscribed to Gemini Advanced) yesterday, even though my last query is weeks past.

Possibly there's a capacity shortage (I'd presume it all runs on the same Google hardware in the end), and they are prioritizing paid inference?

Someone on r/LocalLLaMa shared this link: https://aistudio.google.com/prompts/new_chat?model=gemma-3n-...

https://www.reddit.com/r/LocalLLaMA/comments/1kr8s40/comment...

I generally assume this is how existing model developers have been improving. Gemini especially is crazy fast, and thanks to google search integration, Gemini-the-model doesn’t need to “know” any trivia.

Not a bad idea for next generation models, especially since the state of the art already uses Mixture of Experts.