GAN Math (2020)

Paper authors (and this posts author apparently) like to throw in lots of scary-looking maths to signal that they are smart and that what they are doing has merit. The Reinforcement Learning field is particularly notorious for doing this, but it's all over ML. Often it is not on purpose, everyone is taught this is the proper "formal" way to express these things, and that any other representation is not precise or appropriate in a scientific context.

In practice, when it comes down to code, even without higher-level libraries, it is surprisingly simple, concise and intuitive.

Most of the math elements used have quite straightforward properties and utility, but of course if you combine them all together into big expressions with lots of single-character variables, it's really hard to understand for everyone. You kind of need to learn to squint your eyes and understand the basic building-blocks that the maths represent, but that shouldn't be necessary if it wasn't obfuscated like this.

It takes a while to get into, just like with everything determination is key

Also there are libraries that abstract away most if not all the things, so you don't have to know everything

Haha, I was just going to say the same. I was hoping, I guess naively, that this would explain the math. Not just show me math. While I love a good figure, I like pseudocode just as much :)

Whilst it's maybe not worth studying them in detail I'd say being aware of their existence and roughly how they work is still useful. Seeing the many varied ways people have done things with neural networks can be useful inspiration for your own ideas and perhaps the ideas and techniques behind GANs will find a new life or a new purpose.

Yes you can just concentrate on the latest models but if you want a better grounding in the field some understanding of the past is important. In particular reusing ideas from the past in a new way and/or with better software/hardware/datasets is a common source of new developments!

GAN is not a architecture its a training method. As the models themselves change underneath, GAN remain relevant. (Just as you see autoencoder still being used as a term in new published works, which is even older.)

Though if you can rephrase the problem into a diffusion it seems to be prefered these days. (Less prone to mode collapse)

Gan is famously used for generative usecases, but has wide uses for creating useful latent spaces with limited data, and show up in few-shot-learning-papers. (Im actually not that up to speed on the state of art in few-shot so mabie they have something clever that replace it)

Whenever someone says this I like to point out that they are very often used to train the VAE and VQVAE models that LDM models use. Slowly diffusion is encroaching on its territory with 1-step models, however, and there are now alternative methods to generate rich latent spaces and decoders too, so this is changing, but I'd say up until last year most of the image generators still used an adversarial objective for the encoder-decoder training. This year, not sure..

The article is from 2020, so it would be closer to relevancy back then.

Yeah, title needs (2020) added.

GANs were fun though. :)

Adversarial loss is still use on most image generators, diffusion/autoregressive models work on a latent space (they don't have to, but it would incredibly inefficient) created by an autoencoder, these autoencoders are trained on several losses, usually L1/L2, LPIPS and adversarial.