Prediction Games

I have a hard time keeping up with the literature on this and it's not exactly my area of research, but the "overfitting is ok" always seemed off and handwavy to me. It violates some pretty basic information-theoretic literature, for one thing.

I guess it seems like parameters need to be "counted" differently or there's something misunderstood about what a parameter is, or whether and how it's being adjusted for somewhere. Some of the gradient descent literature I've read, makes it seem like there are sometimes adjustments for parameters as part of the optimization process, so talking about "overfitting doesn't mean anything" is misleading.

It just seems like something where there's a lot of imprecision in terms that is critically important, no definitive explanations for anything, and so forth.

The results are the results, but then again we have hallucinations and weird adversarial probe glitches suggestive of overfitting (see also e.g., http://proceedings.mlr.press/v119/rice20a). I might even suggest the definition of overfitting in a DL context has been poorly operationalized. Sure you can have a training and a test set, but if the test set isn't sufficiently differentiated from the training set, are you going to identify overfitting? I can take training and test sets with a traditional statistical model and if I define the test set a certain way, minimize overfitting results.

I guess I just feel like a lot of overfitting discussions tend to feel kind of handwavy or misleading and I wish they were different. The number of parameters has never really been the correct metric when talking about overfitting, it just happens to align nicely with the correct metric in conventional models.

How many $1 million prizes were given out?

>This is a bitter lesson about the interplay between techlash activism and big tech power structures. Twenty years of privacy complaints have only made tech companies more powerful.

So we should have done what, exactly, Ben?