The Math Behind GANs

Summary

Generative Adversarial Networks refer to a family of generative models that seek to discover the underlying distribution behind a certain data generating process. This distribution is discovered through an adversarial competition between a generator and a discriminator. As we saw in an earlier introductory post on GANs, the two models are trained such that the discriminator strives to distinguish between generated and true examples, while the generator seeks to confuse the discriminator by producing data that are as realistic and compelling as possible. In this post, we’ll take a deep dive into the math behind GANs. My primary source of reference is Generative Adversarial Nets by Ian Goodfellow, et al. It is in this paper that Goodfellow first outlined the concept of a GAN, which is why it only makes sense that we commence from the analysis of this paper. Let’s begin! Motivating the Loss Function GAN can be seen as an interplay between two different models: the generator and the discriminator. Therefore, each model will have its own loss function. In this section, let’s try to motivate an intuitive understanding of the loss function for each. Notation To minimize confusion, let’s define some notation that we will be using throughout this post. \[\begin{multline} \shoveleft x: \text{Real data} \\ \shoveleft z: \text{Latent vector} \\ \shoveleft G(z): \text{Fake data} \\ \shoveleft D(x): \text{Discriminator's evaluation of real data} \\ \shoveleft D(G(z)): \text{Discriminator's evaluation of fake data} \\ \shoveleft \text{Error}(a, b): \text{Error between } a \text{ and } b\\ \end{multline}\] The Discriminator The goal of the discriminator is to correctly label generated images as false and empirical data points as true. Therefore, we might consider the following to be the loss function of the discriminator: \[L_D = \text{Error}(D(x), 1) + \text{Error}(D(G(z)), 0) \tag{1}\] Here, we are using a very generic, unspecific notation for $\text{Error}$ to refer to some func...