ETH Zurich and EPFL to release a LLM developed on public infrastructure

Imo, a lot of the magic is also dataset driven, specifically the SFT and other fine tuning / RLHF data they have. That's what has separated the models people actually use from the also-rans.

I agree with everything you say about getting the experience, the infrastructure is very important and is probably the most critical part of a sovereign LLM supply chain. I would hope there will also be enough focus on the data, early on, that the model will be useful.

The infra does become pretty complex to get a SOTA LLM trained. People assume it's as simple as loading up the architecture and a dataset + using something like Ray. There's a lot that goes into designing the dataset, the eval pipelines, the training approach, maximizing the use of your hardware, dealing with cross-node latency, recovering from errors, etc.

But it's good to have more and more players in this space.

When I read "from scratch", I assume they are doing pre-training, not just finetuning, do you have a different take? Do you mean it's normal Llama architecture they're using? I'm curious about the benchmarks!

Is there not yet a Source where the web has already been scraped and souped down to just the text? It would seem someone would have created such a thing in order to save LLM training from having to reinvent the wheel.

I understand the web is a dynamic thing but still it would seem to be useful on some level.

No performance degradation on training metrics except for the end user. At the end of the day users and website owners have completely orthogonal interests. Users want answers and content, website owners want attention so they can upsell/push ads. You can only serve one master.

I think that the Allen Institute for Artificial Intelligence OLMo models are also completely open:

OLMo is fully open

Ai2 believes in the power of openness to build a future where AI is accessible to all. Open weights alone aren’t enough – true openness requires models to be trained in the open with fully open access to data, models, and code.

https://allenai.org/olmo

Smollm is also completely open as far as I know

> The model will be fully open: source code and weights will be publicly available, and the training data will be transparent and reproducible

This leads me to believe that the training data won’t be made publicly available in full, but merely be “reproducible”. This might mean that they’ll provide references like a list of URLs of the pages they trained on, but not their contents.

Yup, it’s not a dataset packaged like you hope for here, as it still contains traditionally copyrighted material

It's a university, teaching the 'how it's done' is kind of the point

The model will be released in two sizes — 8 billion and 70 billion parameters [...]. The 70B version will rank among the most powerful fully open models worldwide. [...] In late summer, the LLM will be released under the Apache 2.0 License.

We'll find out in September if it's true?

Yes but it’s a very complicated service to deliver. Even if they train great models, they likely will not operationalize them for inference. Those will still be private actors, and the incentives to enshittify will be the same. Also, for AI generally the incentives is much higher than last tech generation, due to cost of running these things. Basically, the free services where you’re the product must aggressively extract value out of you in order to make a profit.

I literally cant fault this, even steelmanning anti AI positions. What makes you say that?

The announcement was at the International Open-Source LLM Builders Summit held this week in Switzerland. Is it so strange that they announced what they are doing and the timeline?

Funding? Deeply biasing European uses to publicly-developed European LLMs (or at least not American or Chinese ones) would make a lot of sense. (Potentially too much sense for Brussels.)

The cliché (at least on my side of the Alps) is that people in Switzerland like to take theiiiir tiiiime.