Teuken-7B-Base and Teuken-7B-Instruct: Towards European LLMs (2024)

Someone apparently did observe ChatGPT (I think it was ChatGPT) switch to Chinese for some parts of it's reasoning/calculations and then back to English for the final answer. That's somehow even weirder than the LLM giving different answers depending on the input.

I suspect this also happens in programming languages. Subjectively I get the feeling that LLMs prefer to write in Python or JS.

Would be interesting to see whether they actually score better in leetcode questions when using python.

Indeed. I've thought from the beginning that LLMs should focus specifically on ONE language for this exact reason (i.e. mediocre/bad duplication of data in multiple languages). All other languages than English essentially "syphon" off capacity/layers/weights that could otherwise have held more genuine data/knowledge. Other languages should not come into the picture afaics - dedicated translation LLMs/existing-solutions can handle this aspect just fine and there's just no salient reason to fold partial-multi-language-capacity in through fuzzy/unorganised training.

Given the fact that LLMs like most neural networks work by passing their input through layers, wouldn't this be expected? There's no going back to an earlier layer and if the first layers are in some sense needed for "translating" [0] to English, any other functionality in those layers cannot be used.

[0] I am simplifying here, but it would make sense for an LLM to learn this, even though the intermediate representation is not exactly English, given the fact that much of the internet in English and the empirical fact that they are good at translating.

For most low-resource languages, support in LLMs is trained through translation pairs between english and the other languages, because translation data is easier to come across than say, conversations about coding, history, physics, basically the kind of data that is usually used for instruct training.

This kind of training data typically looks like ChatGPT style conversations where all the prompts are all templated like “Translate the following text from X to Y: [text]” and the LLM’s expected answer is the translated text.

LLMs can generalize through transfer learning (to a certain extent) from these translation pairs to some understanding (strong) and even answering (weak) in the target language. It also means that the LLM’s actual sweet spot is in translation itself since that’s what was trained in, not just a generalization.

I'd mentally put this in the same box as "chain of thought", where models perform better when explicitly describing the reasoning steps. The only difference in your case being that the model is undertrained in non-English data, so it's "next token prediction" of non-English prompts is less robust, and thus explicitly converting to English and then back makes it better.

This is probably the case for the "deep reasoning" models as well. If you for example try DeepSeek R1, it will likely reason in either English or Chinese (where it presumably is well trained) even if the prompt is in other languages.

I sometimes dream that they would internally reason in Ithkuil and gain amazing precision.

ChatGPT is very informal and talks like a millennial when I ask questions in French. I hate it.

Added. Thanks!

I've just tried it in one of the supported languages, and it seems to respond far better than any model under 24B that I've tried before. With its licensing, it sounds much more exciting to me than the OP.

I live in a country with 3 national languages and I happen to use all of them + English + another one where most of our clients are based. Mistral is the only model atm which doesn’t make a mess of it all. It’s not perfect, but it doesn’t force me to “pretranslate” things.

As a native of another small European language, no state of the art model comes anywhere close to not being laughably bad, so more work in this space is definitely welcomed as far as I'm concerned.

Really? In my experience, Le Chat eventually devolves into spanglish when trying to speak Spanish, so I would have expected worse from Mistral for minority languages.

ah, yes... Europe, the continent with 10 countries

one of them with 50k population

Meltemi is ok, but it's "old" and not that good by today's standards. If you need a good Greek local LLM try https://huggingface.co/ilsp/Llama-Krikri-8B-Instruct. Yes, I know it's based on LLama and not a foundation model, but it is still a LOT better than Meltemi.

You don't think American companies raising hundreds of millions to ten billion for training models contributed to their model performance or market positions?

I think a pile of money and talent is largely the cause of where they're at.

You might be confusing ARC-AGI and EU-ARC which is a language benchmark [1]

[1] https://arxiv.org/pdf/2410.08928

Why would they want more languages from outside of the EU when they've clearly stated they only target the 24 official languages of the European Union?