Deep Learning Is Not So Mysterious or Different

I watched the 3b1b series on neural nets years ago, and it still accounts for 95% of my understanding of AI in general.

I’m not an ML person, but still. That guy has a serious gift for explaining stuff.

His video on the uncertainty principle explained stuff to me that my entire undergrad education failed to!

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

Looks nice - are there written versions?

Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics.

https://work.caltech.edu/telecourse

[dead]

[dead]

I have followed a fair few StatQuest and other videos (treadmills with Youtube are great for fitness and learning in one)

I find that no single source seems to cover things in a way that I easily understand, but cumulatively they fill in the blanks of each other.

Serrano Academy has been a good source for me as well. https://www.youtube.com/@SerranoAcademy/videos

The best tutorials give you a clear sense that the teacher has a clear understanding of the underlying principles and how/why they are applied.

I have seen a fair few things that are effectively.

'To do X, you {math thing}' While also creating the impression that they don't understand why {math thing} is the right thing to do, just that {math thing} has a name and it produces the result. Meticulously explaining the minutiae of {math thing} substitutes for a understanding of what it is doing.

It really stood out to me when looking at UMAP and seeing a bunch of things where they got into the weeds in the math without explaining why these were the particular weeds to be looking in.

Then I found a talk by Leland McInnes that had the format.

{math thing} is a tool to do {objective}. It works, there is a proof, you don't need to understand it to use the tool but the info for that is over there if you want to tale a look. These are our objectives, let's use these tools to achieve them.

The tools are neither magical black boxes, nor confused with the actual goal. It really showed the power of fully understanding the topic.

Also would like to add that he has a YouTube channel as well https://youtube.com/@statquest

Double Bam

I'm a huge fan of HN just for replies such as this that smash the OP's post/product with something better. It's like at least half the reason I stick around here.

Thanks for the great read.

Statistical mechanics is the lens that makes most sense to me. Plus it's well studied.

Good read, thanks for sharing

Here is an example for data-efficient vision transformers: https://arxiv.org/abs/2401.12511

Vision transformers have a more flexible hypothesis space, but they tend to have worse sample complexity than convolutional networks which have a strong architectural inductive bias. A "soft inductive bias" would be something like what this paper does where they have a special scheme for initializing vision transformers. So schemes like initialization that encourage the model to find the right solution without excessively constraining it would be a soft preference for simpler solutions.

You're correct, and the term you're looking for is "regularisation".

There are two common ways of doing this: * L1 or L2 regularisation: penalises models whose weight matrices are complex (in the sense of having lots of large elements) * Dropout: train on random subsets of the neurons to force the model to rely on simple representations that are distributed robustly across its weights

The solution to the L1 regularization problem is actually a specific form of the classical ReLU nonlinearity used in deep learning. I’m not sure if similar results hold for other nonlinearities, but this gave me good intuition for what thresholding is doing mathematically!

I'm not a guru myself, but I'm sure someone will correct me if I'm wrong. :-)

The usual approach to supervised ML is to "invent" the model (layers, their parameters) or more often copy one from known good reference, then define the cost function and feed it data. "Deep" learning just means that instead of a few layers you use a big number of them.

What you describe sounds like an automated way of tweaking the architecture, IIUC? Never done that, usually the cost of a run was too high to let an algorithm do that for me. But I'm curious if this approach is being used?

Yeah, it's straightforward to reproduce the results of the paper whose conclusion they criticize, "Understanding deep learning requires rethinking generalization", without any (explicit) regularization or anything else that can be easily described as a "soft preference for simpler solutions".

The scaling is brutal. If you have a 20k word vocabulary and want to do 3 grams, you need a 20000^3 matrix of elements (8 trillion). Most of which is going to be empty.

GPT and friends cheat by not modeling each word separately, but a large dimensional “embedding” (just a vector if you also find new vocabulary silly). The embedding represents similar words near each other in this space. The famous king-man-queen example. So even if your training set has never seen “The Queen ordered the traitor <blank>”, it might have previously seen, “The King ordered the traitor beheaded”. The vector representation lets the model use words that represent similar concepts without concrete examples.

There is some recent work [0] that explores this idea, scaling up n-gram models substantially while using word2vec vectors to understand similarity. Used to compute something the authors call the Creativity Index [1].

[0]: https://infini-gram.io [1]: https://arxiv.org/abs/2410.04265v1

this is pretty close to how language models worked in the 90s-2000s. deep language models -- even GPT 2 -- are much much better. on the other hand, the n-gram language models are "surprisingly good" even for small n.

The problem is that for any reasonable value of N (>100) you will need prohibitive amounts of storage. And it will be extremely sparse. And you won’t capture any interactions between N-99 and N-98.

Transformers do that fairly well and are pretty efficient in training.

> How close would this be to GPT 2

Here's a post from 2015 doing something a bit like this [1]

[1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139

Pretty sure this wouldn't produce anything useful. Pretty sure this would generate incoherent gibberish that looks and sounds like English but makes no sense. This ignores perhaps the most important element of LLM's, the attention mechanism.

Markov chains are very very far off from gpt2.

In a sense; linear regression can be computed exactly so refers to a specific technique for producing a linear model.

Most artificial neurons are trained stochastically rather than holistically, i.e. rather than looking at the entire training set and computing the gradient to minimize the squared loss or something similar, they look at each training example and compute the local gradient and make small changes in that direction.

In addition, the "activation function" almost universally used now is the rectified linear unit, which is linear for positive input and zero for negative input. This is non-decreasing at least as a function, but the fact that it is not monotonic means that there is no additional loss accrued for overcorrecting in the negative direction.

Given this, using the term "linear regression" to describe the model of an artificial neuron is not really a useful heuristic.

This is like saying "the human brain is just some chemistry." You have the general idea correct, but there's a lot more going on that just that, and the emergent system is so much more complex that it deserves its own separate field.

MLPs are compositions of generalized linear models. That's not very enlightening though; the "mysterious" part is the macroscopics of the composition.

Please formulate your critique instead of simply labeling it with negative words.

It's a higher quality article than half the submissions to NeuroIPS and the rest of the AI/ML conferences, because it has potential to remain relevant next year, and because of its high didactic content. I wouldn't classify it as a "hot take".

The mystery was never in the "how do computers calculate the probabilities of next tokens" but rather in the "why is it able to work so well" and "what does this individual neuron contribute to the whole model"

You're misunderstanding. A level of abstraction is necessary for operation of modern systems. There is no human alive who, given an intermediate step in the middle of some running learning algorithm, is able to understand and mentally model the full system at full man-made resolution, that is, down to the transistor level, on a modern CPU. Someone wishing to understand a piece of software in 2025 is forced to, at some point, accept that something somewhere "does what it says on the tin" and model it thusly rather than having a full understanding.

But the weights trained from machine learning are a black box, in the sense that no human designed e.g. the image processing kernels that those weights represent.

That is one reason people are skeptical of them, not only is training a large model at home expensive, not only is the data too big to trivially store, but the weights are not trivial to debug either