Mistral Small 3

Structured output is solved, it is structuring data that's not, because that is an unbounded problem. There is no limit to how messy your data may be, and no limit to the accuracy and efficiency you may require.

I have used such models to structure human-generated data into sth a script can then read and process, getting important aspects in this data (eg what time the human reported doing X thing, how long, with whom etc) into like a csv file with columns eg timestamps and whatever variables I am interested in.

For anyone who thinks it isn't "solved", outlines debunked the paper which claims that "structured generation harms creativity":

https://blog.dottxt.co/say-what-you-mean.html

And for anyone looking to dig deeper, check out "grammar-based sampling."

What’s the “assistant: {" trick? You just end your prompt with that?

Most of them are experimental studies. So it would be text extraction of something like title, authors, species of the study, sample size etc. And classify based on the content of the pdfs.

I tried the GPT-4o, it's good but it'll cost a lot if I want to process all the documents.

Cool, thanks for the reply. Looking forward to following along with your progress.

A man walks up to an llm and asks it to tell him a joke about a puffin and a badger.

The llm replies with a joke that is barely a joke.

The man says "another."

The llm gives another unfunny response.

"Another!"

Followed by another similarly lacking response.

"Another!"

With exasperation, the llm replies "stop badgering me!"

Except it won't, because that's not a high likelihood output. ;)

And yet LLMs tend to in fact be very funny, just very very rarely on purpose.

> The insistence every time open sourced model weights come up that it is not "truly" open source is tiring. There is enormous value in open source weights compared to closed APIs. Let us call them open source weights. What you want can be "open source data" or somesuch.

Agree that there is more value in open source weights than closed APIs, but what I really want to enable, is people learning how to create their own models from scratch. FOSS to me means being able to learn from other projects, how to build the thing yourself, and I wrote about why this is important to me here: https://news.ycombinator.com/item?id=42878817

It's not a puritan view but purely practical. Many companies started using FOSS as a marketing label (like what Meta does) and as someone who probably wouldn't be a software developer without being able to learn from FOSS, it fucking sucks that the ML/AI ecosystem is seemingly OK with the term being hijacked.

Agree that it's not exactly the same, all analogies have holes, they're simplifications after all.

I guess I'm vary of the messaging because I'm a developer 99% thanks to FOSS, and being able to learn from FOSS projects how to build similar stuff myself. Without FOSS, I probably wouldn't have been able to "escape" the working-class my family was "stuck in" when I grew up.

I want to do whatever I can to make sure others have the same opportunity, and it doesn't matter if the weights themselves are FOSS or not, others cannot learn how to create their own models based on just looking at the weights. You need to be able to learn the model architecture, training and what datasets models are using too, otherwise you won't get very far.

> This kind of purity test mindset doesn't help anyone. They are shipping the most modifiable form of their model.

It does help others who might be stuck in the same situation I was stuck in, that's not nothing nor is it about "purity". They're not shipping the most open model they can, they could have done something like OLMo (https://github.com/allenai/OLMo) which can teach people how to build their own models from scratch.

Depending on format, they might.

The same is true for FOSS in general. You're arguing that because no one (almost) builds their own X from scratch, there is therefore no value in having resources available for how to build your own X from scratch.

For someone who basically couldn't become a developer with FOSS, this way of thinking is so backwards, especially on Hacker News. I thought we were pro-FOSS in general, but somehow LLMs get a pass because "they're too complicated and no one would build one from scratch".

Like every other Open Source LLM weights, yes. But looking around, there are models that are 100% FOSS, like OLMo (https://github.com/allenai/OLMo).

Also, I don't buy the argument that because many in the ecosystem mislabel/mislead people about the licensing, makes it ethically OK for everyone else to do so too.

https://github.com/huggingface/open-r1

Has that actually been tried in court, or is that your guess? Because you seem confident, but I don't think this has been tried (yet)

AMD could arguably do it. But they have to focus to stay above water at all, and "put 128GB or more of DDR5 ram on any previous-gen GPU" is probably not in their focus. With the state of their software it's not even certain if the community could pick up the slack and turn that into a popular solution.

Nvidia upcoming 'minipc' has shared ram up to 128gb for around 3k. No a competitor but pretty good for enthusiast.

Hopefully is at least quadchannel.

> Because memory bandwidth is the #1 bottleneck for inference, even more than capacity.

But there are a ton of models I can't run at all locally due to VRAM limitations. I'd take being able to run those models slower. I know there are some ways to get these running on CPU orders of magnitude slower, but ideally there's some sort of middle ground.

You might find this comparison chart helpful: https://www.airtrain.ai/blog/how-15-top-llms-perform-on-clas...

Note: from October; also I work at Airtrain

Historically the problem with using LLMs for the super simple conventional NLP stuff is that they were hard to control in terms of output. If you wanted a one-word answer for a classification task, you'd often have to deal with it responding in a paragraph. This obviously hurts precision and accuracy quite a bit. There were tricks you could use (like using few-shot examples or GBNF grammars or training low-rank adapters or even re-asking the model) to constrain output a bit, but they weren't perfect.

Over the last 12-18 months though, the instruction-following capabilities of the models have improved substantially. This new mistral model in particular is fantastic at doing what you ask.

My approach to this personally and professionally is to just benchmark. If I have a classification task, I use a tiny model first, eval both, and see how much improvement I'd get using an LLM. Generally speaking though, the vram costs are so high for the latter that its often not worth it. It really is a case-by-case decision though. Sometimes you want one generic model to do a bunch of tasks rather than train/finetune a dozen small models that you manage in production instead.

Super easy to get started, but lacking for larger datasets where you want to understand a bit more about predictions. You generally lose things like prediction probability (though this can be recovered if you chop the head off and just assign output logits to classes instead of tokens), repeatability across experiments, and the ability to tune the model by changing the data. You can still do fine tuning, though itll be more expensive and painfaul than a BERT model.

Still, you can go from 0 to ~mostly~ clean data in a few prompts and iterations, vs potentially a few hours with a fine tuning pipeline for BERT. They can actually work well in tandem to bootstrap some training data and then use them together to refine your classification.