Open Euro LLM: Open LLMs for Transparent AI in Europe
33 comments
·February 3, 2025jpdus
FanaHOVA
The problem is that: - These are not really super computing cluster in LLM terms. Leonardo is a 250 PFlops cluster. That is really not much at all. - If people in charge of this project actually believe R1 costs $5.5M to build from scratch, it's already over.
sarusso
They allocated €37.4 million [1]. As an European, I truly don’t understand why they keep ignoring that the money required for such projects is at least an order of magnitude more.
[1] https://digital-strategy.ec.europa.eu/en/news/pioneering-ai-...
esperent
Deepseek's release has shown that there's no great risk in getting left behind. All the info is out there, people with skills are readily available, creating a model that will match whatever current model is considered frontier level is not that hard for an entity like the EU.
For everyone here shouting that the EU needs to do something, be a leader, what have they lost so far by choosing to lead in legislation instead of development?
They've lost nothing. They've gained a lot.
They can use the same frontier level open source model as everyone else, and meanwhile, they can stay on top of harmful uses like social or credit scoring.
Also speaking as a European, legislation is kind of the point of a government in the first place. I do think the EU goes too far in many cases. But I haven't seen anything that makes me think they're dealing with this particular hype train badly so far. Play the safe long game, let everyone else spend all the money, see what works, focus on legislation of potentially dangerous technology.
tesch1
> lead in legislation
> legislation is kind of the point of a government
As an American, most of this post reads like doublespeak satire. I guess it's not, but just to put a transatlantic pov here.
I'll add a sports metaphor for good measure: in order to become expert football players, we'll get tickets to watch the best teams play.
esperent
> As an American, most of this post reads like doublespeak satire
Yeah, you guys have a lot of brainwashing to get over. I can imagine that you're deeply conditioned to read any outside views on politics as satire.
One kind of brainwashing is the need to reframe everything political into sports metaphors. The EU is not a sports team. It's a political entity. Whatever you might have been taught, these are very different things, with different needs. You can't have meaningful conversations about a political entity via sports metaphors.
Well, maybe in US politics you can. There you have two teams determined to beat the other at all costs. EU politics isn't like that. We are trying to work together, not kill each other.
null
Etheryte
Personally I'm rather happy that the allocation was not too large at first, even that is quite a sizeable sum. The EU is great at kickstarting projects that sound like a panacea, but end up not leading to anything. Once they have something to show, by all means, throw more money at them.
nradov
The trap that these EU projects typically fall into is that they burn all of the grant funding on paying politically connected consultants to write reports. No one gets around to building an MVP.
everyone
Not anymore, with Deepseek's stuff right? Which is open.
sarusso
DeepSeek had plenty of R&D expertise which were not included in the (declared) model training cost. Here we are talking about building something nearly from scratch, even if there is an open source starting point you still need the infrastructure, expertise and people to make it work, which with that budget are going to be hard to secure. Moreover these projects take months and months to get approved, meaning that this one was conceived long before DeepSeek, thus highlighting the original disalignment between the goal and the budget. DeepSeek might have changed the scenario (I hope so) but it would be just a lucky ex-post event… not a conscious choice behind that budget.
blackeyeblitzar
DeepSeek probably spent closer to two billion on hardware. And then there’s the energy cost of numerous runs, staff costs, all of that. The 5.5m cost was basically misleading info, maybe used strategically to create doubt in the US tech industry or for DeepSeek’s parent hedge fund to make money off shorts.
null
NunoSempere
They just shipped a frontpage; there is no model yet.
m3kw9
They want transparency but they give small peanuts, are they thinking they can just take llama and then distill other models into it can call it transparent? I’m not sure if they understand how that works
Havoc
Im all for more models. Go for it
Xelbair
No model, so irrelevant because EU is extremely late to the party, as i say that as someone from EU.
ur-whale
> The models will be developed within Europe's robust regulatory framework, ensuring alignment with European values while maintaining technological excellence.
translation: weakest competitor in the contest enters the fight with both hands tied behind its back and a budget akin to what OpenAI spends in a week on compute.
But hey, more power to you Europe: the more models around, the better.
Eventually, we'll be able to bind all those censored models worldwide into one giant mixture of expert to get rid of the built-in censorship of each individual component.
mt_
maybe 5 million for the html frontend
moffkalast
> The models will be developed within Europe's robust regulatory framework, ensuring alignment with European values while maintaining technological excellence
As a European, that's practically an oxymoron. The more one limits oneself to legally clean data, the worse the models will be.
I hate to be pessimistic from the get go, but it doesn't sound like anything useful will be produced by this and we'll have to keep relying on Google to do proper multilinguality in open models because Mistral can't be arsed to bother beyond French and German.
Fnoord
I've been using Mistral past week due to changes in geopolitics, and Mistral works absolutely great in English. I haven't bothered in my native language yet, but in English it worked great. Better than my first experience with ChatGPT (GPT 3.5), actually.
lnauta
I've been using mistral for most of January at the same rate as chatgpt before. I decided to pay for it as its per token (in and out) and the bill came yesterday... A whopping 1 cent. Thats probably rounded up.
htrp
>Mistral can't be arsed to bother beyond French and German.
Any more details here or a writeup you can link to?
moffkalast
My own experience mainly, only Gemma seems to have been any good for Slavic languages so far, and only the 27B when unquantized is reliable enough to be in any way usable.
Ravenwolf posts tests on his German benchmarks every so often in locallama and most models seem to do well enough, but I've heard some claims from people about Mistral's being their favorite models in German anyhow. And I think Mistral-Large scores higher than Llama-405B in French on lmsys and that's at least something one would expect from a French company.
jampekka
What do you mean by relying on Google?
Llama 3.1 and DeepSeek v3/R1 largest models are rather good at even a niche language like Finnish. The performance does plummet in the smaller versions, and even quantization may harm multilinguality disproportionally.
Something like deliberately distilling specific languages from the largest models could work well. Starting from scratch with a "legal" dataset will most likely fail as you say.
Silo AI (co-lead of this model) already tried Finnish and Scandinavian/Nordic models with the from-scratch strategy, and the results are not too encouraging.
moffkalast
Yes I think small languages which have a total corpus of maybe a few hundred million tokens total have no chance of producing a coherent model without synthetic data. And using synthetic data from existing models trained on all public (and less public) data is enough of a legally gray area that I wouldn't expect this project to consider it, so it's doomed before it even starts.
Something like 4o is so perfect in most languages that one could just make an infinite dataset from it and be done with it. I'm not sure how OAI managed it tbh.
simion314
>As a European, that's practically an oxymoron. The more one limits oneself to legally clean data, the worse the models will be.
Train an LLM with text books and other legal books, you do not need to train it on pop culture to make it intelligent.
For face generations you might need to be more creative, you should not need milions of images stolen from social media to train your model.
But makes sense that tech giants do not want to share their data set and be transparent about stuff.
jampekka
> Train an LLM with text books and other legal books
Without licenses to the books, they are just as illegal (and maybe even moreso) than web content.
troyvit
If LLM organizations are free to throw billions at hardware they can spare a paltry €50 million for 10 million e-books though, right?
idunnoman1222
Open AI fed their original model Anna’s archive for breakfast.
As someone who is in general skeptical of programs like this (and an European) there are 2 remarkable / timely things about this:
- This project doesn't just allocate money to universities or one large company, but includes top research institutions as well as startups and GPU time on supercomputing clusters. The participants are very well connected (e.g. also supported by HF, Together and the likes with European roots) - Deepseek has just shown that you probably can't beat the big labs with these resources, but you can stay sufficient close to the frontier to make a dent.
Europe needs to try this. Will this close the Gap to the US/China? Probably not. But it could be a catalyst for competitive Open source models and partially revitalize AI in Europe. let's see..
PS: on Twitter there was a screenshot yesterday that in a new EU draft, "accelerate" was used six times. Maybe times are changing a little bit.
Disclaimer: Our company is part of this project, so I might be biased.