Yann LeCun to depart Meta and launch AI startup focused on 'world models'

Maybe it's time for Bell Labs 2?

I guess everyone is racing towards AGI in a few years or whatever so it's kind of impossible to cultivate that environment.

More importantly even if you do want it, and there are business situations that support your ambitions. You still have to do get into the managerial powerplay, which quite honestly takes a separate kind of skill set, time and effort. Which Im guessing the academia oriented people aren't willing to do.

Its pretty much dog eat dog at top management positions.

Its not exactly a space for free thinking timelines.

Meta did exactly that, kept him but reduced his scope. Did the broader research community benefit from his research? Absolutely. But did Meta achieve a good outcome? Probably not.

If you follow LeCun on social media, you can see that the way FAIR’s results are assessed is very narrow-minded and still follows the academic mindset. He mentioned that his research is evaluated by: "Research evaluation is a difficult task because the product impact may occur years (sometimes decades) after the work. For that reason, evaluation must often rely on the collective opinion of the research community through proxies such as publications, citations, invited talks, awards, etc."

But as an industry researcher, he should know how his research fits with the company vision and be able to assess that easily. If the company's vision is to be the leader in AI, then as of now, he seems to have failed that objective, even though he has been at Meta for more than 10 years.

LeCun was always part of FAIR, doing research, not part of the LLM/product group, who reported to someone else.

I believe that the fact that Chinese models are beating the crap of of Llama means it's a huge no.

None of Meta's revenue has anything to do with AI at all. (Other than GenAI slop in old people's feeds.) Meta is in the strange position of investing very heavily in multiple fields where they have no successful product: VR, hardware devices, and now AI. Ad revenue funds it all.

Agreed, I am surprised he is happy to stay this long. He would have been on paper a far better match at a place like pre-Gemini-era Google

I politely disagree - it is exactly an industry researcher's purpose to do the risky things that may not work, simply because the rest of the corporation cannot take such risks but must walk on more well-trodden paths.

Corporate R&D teams are there to absorb risk, innovate, disrupt, create new fields, not for doing small incremental improvements. "If we know it works, it's not research." (Albert Einstein)

I also agree with LeCun that LLMs in their current form - are a dead end. Note that this does not mean that I think we have already exploited LLMs to the limit, we are still at the beginning. We also need to create an ecosystem in which they can operate well: for instance, to combine LLMs with Web agents better we need a scalable "C2B2C" (customer delegated to business to business) micropayment infrastructure, because as these systems have already begun talking to each other, in the longer run nobody would offer their APIs for free.

I work on spatial/geographic models, inter alia, which by coincident is one of the direction mentioned in the LeCun article. I do not know what his reasoning is, but mine was/is: LMs are language models, and should (only) be used as such. We need other models - in particular a knowledge model (KM/KB) to cleanly separate knowledge from text generation - it looks to me right now that only that will solve hallucination.

LLMs and Diffusion solve a completely different problem than world models.

If you want to predict future text, you use an LLM. If you want to predict future frames in a video, you go with Diffusion. But what both of them lack is object permanence. If a car isn't visible in the input frame, it won't be visible in the output. But in the real world, there are A LOT of things that are invisible (image) or not mentioned but only implied (text) that still strongly affect the future. Every kid knows that when you roll a marble behind your hand, it'll come out on the other side. But LLMs and Diffusion models routinely fail to predict that, as for them the object disappears when it stops being visible.

Based on what I heard from others, world models are considered the missing ingredient for useful robots and self-driving cars. If that's halfway accurate, it would make sense to pour A LOT of money into world models, because they will unlock high-value products.

> but an enterprise like Facebook can't have their top researcher work on risky things when there's surefire paths to success still available.

Bell Labs

>the huge impact they're already having

In the software development world yes, outside of that, virtually none. Yes, you can transcribe a video call in Office, yes, but that's not ground breaking. I dare you to list 10 impacts on different fields, excluding tech and including at least half blue collar fields and at least half white collar fields , at different levels from the lowest to the highest in the company hierarchy, that LLM/Diffusion models are having. Impact here specifically means a significant reduction of costs or a significant increase of revenue. Go on

He's quoted in OP as calling them 'useful but fundamentally limited'; that seems correct, and not at all like he's denying their utility.

Unless I've missed a few updates, much of the JEPA stuff didn't really bear a lot of fruit in the end.

While I agree with your point, “Superintelligence” is a far cry from what Meta will end up delivering with Wang in charge. I suppose that, at the end of the day, it’s all marketing. What else should we expect from an ads company :?

Hard to tell.

The last time LeCun disagreed with the AI mainstream was when he kept working on neural net when everyone thought it was a dead end. He might be entirely right in his LLM scepticism. It's hardly a surefire path. He didn't prevent Meta from working on LLM anyway.

The issue is more than his position is not compatible with short term investors expectations and that's fatal in a company like Meta at the position LeCun occupies.

You missed an opportunity to use paperclips instead of toothpicks, as your example.

Would be very inline with the AI angle.

How do LLMs help with clicks and attention minutes? Why do they spend $100+B a year in AI capex, more than Google and Microsoft that actually rent AI compute to clients? What are they going to do with all that compute? It’s all so confusing

Sad to hear it has come to attention minutes, used to be seconds.

If LeCun's research has made Meta a powerhouse of video generation or general purpose robotics - the two promising directions that benefit from working with visual I/O and world modeling as LeCun sees it - it could have been a justified detour.

But that sure didn't happen.

LLMs get results, yes. They are getting adopted, and they are making money.

Frontier models are all profitable. Inference is sold with a damn good margin, and the amounts of inference AI companies sell keeps rising. This necessitates putting more and more money into infrastructure. AI R&D is extremely expensive too, and this necessitates even more spending.

A mistake I see people make over and over again is keeping track of the spending but overlooking the revenue altogether. Which sure is weird: you don't get from $0B in revenue to $12B in revenue in a few years by not having a product anyone wants to buy.

And I find all the talk of "non-deterministic hallucinatory nature" to be overrated. Because humans suffer from all of that too, just less severely. On top of a number of other issues current AIs don't suffer from.

Nonetheless, we use human labor for things. All AI has to do is provide a "good enough" alternative, and it often does.

OpenAI and Anthropic are making north of 4B/year revenue so some companies have figured out the money making part. ChatGPT has some 800M users according to some calculations. Whether it's enough money today, enough money tomorrow, is of course a question but there is a lot of money. Users would not use them in a scale if they do not solve their problems.

If you hire a house cleaner to clean your house, and the cleaner didn't do well, would you eject yourself out of the house? You would not. You would change to a new cleaner.

Sure, but that "someone else" is the man writing the checks. If the roles were reversed, he'd be the one being fired now.

Who are you referring to?

Hopefully one day, in a galaxy far far away, someone builds something on those foundations.

What is responsibility if you can afford good lawyers?

Learning from the real world, including how it responds to your own actions, is the only way to achieve real-world competency, intelligence, reasoning and creativity, including going beyond human intelligence.

The capabilities of LLMs are limited by what's in their training data. You can use all the tricks in the book to squeeze the most out of that - RL, synthetic data, agentic loops, tools, etc, but at the end of the day their core intelligence and understanding is limited by that data and their auto-regressive training. They are built for mimicry, not creativity and intelligence.

So... that seems like possible path towards AGI. Doesn't it?

Isn't this idea demonstrably false due to the existence of various sensory disorders too?

I have a disorder characterised by the brain failing to filter own its own sensory noise, my vision is full of analogue TV-like distortion and other artefacts. Sometimes when it's bad I can see my brain constructing an image in real time rather than this perception happening instantaneously, particularly when I'm out walking. A deer becomes a bundle of sticks becomes a muddy pile of rocks (what it actually is) for example over the space of seconds. This to me is pretty strong evidence we do not experience reality directly, and instead construct our perceptions predictively from whatever is to hand.

> humans have a direct experience of reality not mediated by as Alan Kay put it three pounds of oatmeal

Is he advocating for philosophical idealism of the mind or does he has an alternate physicalist theory?

That way he may get a very good lizard. Getting Einstein though takes layers of abstraction.

My thinking is that such world models should be integrated with LLM like the lower levels of perception are integrated with higher brain function.

Great strawman.

For the same reason I don't attribute those successes to Zuckerberg I don't attribute them to LeCun either.