Jürgen Schmidhuber：the Father of Generative AI Without Turing Award

It's a clash of cultures. He is an academic that cares for understanding where ideas came from. His detractors need to be the smartest people in the room to get paid millions and raise billions.

It's not very sexy to say 'Oh yes, we are just using an old Soviet learning algorithm on better hardware. Turns out we would have lost the cold war if the USSR had access to a 5090.".

It's very common in science for people to have had results they didn't understand the significance of that later were popularized by someone else.

There is the whole thing with Damadian claiming to have invented MRI (he didn't) when the Nobel prize went to Mansfield and Lauterbur (see the Nobel prize part of the article). https://en.m.wikipedia.org/wiki/Paul_Lauterbur

And I've seen other less prominent examples.

It's a lot like the difference between ideas and execution and people claiming someone "stole" their idea because they made a successful business from it.

I think he did understand both the significance of his work and the importance of hardware. His group pioneered porting models to GPUs.

But personal circumstances matter a lot. He was stuck at IDSIA in Lugano, i.e. relatively small and not-so-well funded academia.

He could have done much better in industry, with access to lots of funding, a bigger headcount, and serious infrastructure.

Ultimately, models matter much less than infrastructure. Transformers are not that important, other architectures such as deep SSMs or xLSTM are able to achieve comparable results.

This is wildly unfair.

An academic lab doesn't have the budget to spend billions on building up a new GPU based supercomputer cluster every 3 years.

As an example: the top performing supercomputer of 2005 has the has around 30% less flops than a 5090.

But as far as I understand, Schmidhuber's claim is more severe: namely that Bengio, Hinton and LeCun intentionally failed to cite prior work by others (including himself) but instead only cited each other in order to boost their respective scientific reputation.

I personally think that he's not doing himself or his argument of favor by presenting it the way he does. While he basically argues that science should be totally objective and neutral, there's no denying that if you put yourself in a less likeable light, you're not going to make any friends.

On the other hand, he's gone at length with compiling detailed references to support his points. I can appreciate that because it makes his argument a lot less hand-wavey: you can go to his blog and compare the cited references yourself. Except that I couldn't because I'm not an ML expert.

I can see how someone could feel like that if they looked at the world in a particular way.

I have had plenty of ideas in the last few years that I have played with that I have seen published in papers in the following months. Rather than feeling like "I did it first" I feel gratified that not only was I on the right track, but someone else has done the hard slog.

Most papers are not published by people who had the idea the day before. Their work goes back further than that. Refining the idea, testing it and then presenting the results takes time, sometimes years, occasionally decades.

If this happens to you, don't think "Hey! That idea belongs to me!". Thank them for proving you right.

Now if they patent it, that's a different story. I don't think the ideas that sometimes float through my brain belong to me, but I'm not keen on them belonging to someone else either.

I chuckled but I also maybe didn't understand. Is the joke that computer science is a generalization of math? That can't be rigth.

I'm sure he wouldn't object to a Fields Medal either.

A lot of this is because nobody likes braggers - however, in all fairness, his argument is that a lot of what is considered modern ML is based on many previous results, including but not limited to his own research.