Sharing new research, models, and datasets from Meta FAIR

I hope that Llama 4 or 5 will have a different architecture. All released llamas are +/- same inference with a better training pipeline. The downside is that llamacpp will probably not be able to run new models and maybe it will be too much big rewrite, so we will need new c,cpp,go,rust programs.

Side track, but does anyone have suggestions about how to better present such content. I am struggling with similar docs/demos.

As a documentation page, each section is laid out uniformly with section heading, content, link to code and link to paper.

However the page itself is a blog post which will be difficult to find again next year.

Are there other examples of companies having well presented technical summaries which remain findable from the hime page?

it's a bit ironic that Meta ended up becoming the largest "open ai" org.

all right, yeah, it's not "open source", but hey, it is open to use and they're publishing their research openly as well.

Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But ya, there are many directions to go from here!

Imagine that something fundamental shifts in the world of AI research. It could be anything: AI suddenly makes programmers much more productive, AI becomes very good at identifying vulnerabilities, AI chat becomes a new major source of entertainment, AI images become an item popularly shared on Instagram (etc)

Suppose any one of these things happened and suddenly Facebook wished that it had access to state of the art models so that it could customize them for its uses (internal developers or tools, embedding in their app).

Imagine how they would feel if the only way they could access these models were by signing 7-9 figure deals with a model dealer like OpenAI. Even worse, imagine if one of their main competitors in advertising started providing robust AI tools to help advertisers adapt their creatives to various form factors. Facebook is now way behind and possibly has to shell out millions to a company like OpenAI all while also losing ad market share worth billions per quarter (ads on Google start performing much better, so Google gets more ad spend)

If this worst case scenario came to pass, Facebook would look foolish. If even one of these things were likely their investments make sense. The rest (open source, make meta a cool place to work) are a strategy credit.

“Commoditize you complement” may be a good way of framing it. Consider that if OpenAI succeeds dramatically and is the only game in town, they could extract huge rents for anyone using their service. So it’s in other companies interests (or anyone who wants to use AI) that the AI ecosystem have lots of competition to keep prices low.

You can't have enough top researchers without letting them publish.

Those AI experts are a played a critical role in Meta getting that $70B in the first place

everyone that has responded so far has it wrong (naively so).

FB sells ad space on several apps. those apps needs people on them in order for the ad space to be worth anything. people, in turn, need content to attract them to the apps. so it's simple: enable people/companies/whomever to generate tons of content for cheap and consequently share it on the apps. that's it.

This is a great video - places o1 in context. with openAI, google and meta releases going at it at this pace, anthropic is next up..

That’s a fun idea. I’ve always wondered about experimenting with u-nets and hourglass nets for text data since they’re so efficient at capturing global and local context (in vision, anyway). But I’ve never tried it.

It lets those providing AI video generation services watermark all of their videos. So it isn't intended to by voluntary. You would be left with those services that don't comply with whatever the current Big Tech rules are, like people who used Grok/X.ai to generate images in support of Trump despite Grok/X.ai being inferior. https://arstechnica.com/information-technology/2024/08/musks...

How much does it take to train a model at this point? I’d tend to expect that it’ll be in range of any major state or most oligarchs in the next couple years (if it isn’t already). So, making it is probably best of everybody understands the watermarking to be voluntary. Images and videos aren’t worth the bits they are printed in at this point, as evidence of anything in particular.

It originates in Yann LeCunn’s paper from 2022 [1], the term AMI being district from AGI. However, the A has changed over the past few years from autonomous to advanced and even augmented, depending on context

[1] https://openreview.net/pdf?id=BZ5a1r-kVsf

I think Lecun doesn't like the term AGI.

I'm waiting for when they're called Minds :)

I would guess it’s in response to the recent market studies showing that the general public views anything labeled “AI” as a likely scam and untrustworthy.

Even though Meta doesn't sell I/PaaS, Meta's fitness goes up when AI is in the hands of more players than just Google and OpenAI. Commoditize AI and you create a diverse set of businesses that will reach customers through Meta's platforms.

They still ruin society with the Facebook, no matter how much good they do with LLM.

It's not redeeming if you still continue with the original sin.