Generative Adversarial Network (GAN): Letting the Generator Learn the Data Distribution Against a Dynamic Discriminator

The fundamental question that the Generative Adversarial Network (GAN) must answer is: How does random noise learn to become a real image? In supervised learning, each image is paired with a correct-answer label, and the loss function compares the prediction with the ground truth pixel by pixel, making the gap clearly computable. The generation task has no such label—there is no "standard answer" image that tells you what a given noise should become. So the GAN takes a different path: instead of directly specifying what the output should be, it lets another network judge whether the output "looks real".

The two networks each play their own role. The generator G receives a noise vector z sampled from a simple distribution and maps it through multiple layers of transformation into a sample image, denoted as G(z). The discriminator D receives an image x and outputs a scalar D(x), representing the probability that it judges this image to come from real data. D's training direction is clear: give high scores to real samples and low scores to fake samples generated by G, that is, raise real scores while suppressing fake sample scores. G's training direction depends entirely on D: it adjusts its parameters along D's gradient, moving the output toward "more real" and thereby increasing the score D gives to fake samples. The generator never sees any correct labels from beginning to end; all the direction it obtains comes from the discriminator's scores and gradients.

This design makes the discriminator play two roles at once: it is both a learnable loss function and an opponent that plays against the generator. As a loss, it rewards high-frequency realism far better than a fixed pixel-wise error—pixel-wise loss tends to blur sharp details into a fuzzy average, whereas the discriminator can capture structures such as texture and edges that pixel-level error struggles to characterize. As an opponent, it is also updated along with the generator. Each time the generator becomes a little stronger, the discriminator must relearn a new decision boundary, and the "loss landscape" faced by the generator changes accordingly. This is the core feature of GAN optimization: the objective is not stationary, but a dynamic target that is constantly moving and shaped in real time by the opponent.

To precisely describe the previous section's "you chase, I flee" relationship, we need to answer a more specific question: What exactly are D and G optimizing in opposite directions? The original formulation of the Generative Adversarial Network (GAN) writes it as a two-player minimax objective, where both sides share the same value function but optimize it in opposite directions:

min_G max_D E_data log D(x) + E_z log(1 − D(G(z)))

First look at the discriminator side, that is, max_D. The expectation notation E_data means averaging over the real data distribution, and E_z means averaging over the noise distribution. D wants to push log D(x) higher—for real samples, the closer D(x) is to 1, the larger this term; at the same time, push log(1 − D(G(z))) higher—for generated samples, the closer D(G(z)) is to 0, the larger this term. Taken together, D is maximizing the log-likelihood of correctly distinguishing real and fake samples: real samples judged real, fake samples judged fake, and the more thoroughly it does so, the higher the value function becomes.

Now look at the generator side, that is, min_G. G cannot affect the first term, because G does not participate in it; it can only act on the second term. In the original saturated form, G minimizes log(1 − D(G(z))). When D(G(z)) approaches 0, that is, the discriminator cleanly identifies fake samples as fake, log(1 − D(G(z))) approaches 0, and this term has little room left to decrease; the only way G can lower it is to make D(G(z)) as close to 1 as possible, i.e., to fool the discriminator. So G's objective is exactly opposite to D's: D wants to judge fake samples as fake, and G wants to make fake samples judged as real. This is the meaning of "minimax"—the discriminator first maximizes the value function, and the generator then minimizes it, with the two sides alternating repeatedly.

This saturated form is theoretically clean, but in practice it has a gradient problem. When D is very strong and almost completely sees through fake samples, D(G(z)) is close to 0, and the gradient of the saturated loss also approaches 0, so G can hardly learn anything. Therefore, practical training often uses the non-saturating generator loss −E log D(G(z)). It shares the same optimum as the saturated form—both want to make D(G(z)) approach 1—but the gradient behavior is completely different: when D is stronger and D(G(z)) is smaller, the gradient of −log D(G(z)) is larger, thereby transmitting a stronger learning signal to the generator. In other words, the non-saturating form merely reshapes the gradient curve of the same objective, so that the early-training situation where "the discriminator crushes the generator" no longer freezes the generator's learning.

The loss formula is abstract; plugging in concrete numbers clarifies what each part measures. Suppose at some step the discriminator outputs D(x_real) = 0.9 for a real sample and D(G(z)) = 0.2 for a generated sample. Compute the discriminator loss by minimizing binary cross-entropy: it must simultaneously reward "classifying real samples as real" and "classifying fake samples as fake", so it averages the two terms equally, giving

L_D = −[log 0.9 + log(1 − 0.2)] / 2 ≈ 0.164

First, break it down. −log 0.9 ≈ 0.105; this term penalizes the discriminator for not being confident enough about real samples—0.9 is already close to 1, so the penalty is small. log(1 − 0.2) = log 0.8, and after negating ≈ 0.223; this term penalizes the discriminator for giving the fake sample a score of 0.2, which is still not low enough—0.2 still has distance from 0, so it contributes more. Averaged, 0.164 indicates that the discriminator is generally moving in the right direction but still has room before perfect discrimination.

Now look at the generator. The non-saturating generator loss only cares about "the degree to which fake samples are taken as real", namely −log D(G(z)) = −log 0.2 ≈ 1.609. This value is clearly larger than the discriminator loss, directly showing that the current fake sample is still far from "being judged as real": the discriminator only gave it 0.2, and the generator must pull it close to 1 to succeed, with a huge distance in between, so the loss remains high.

Comparing these two numbers also reveals a point that is easy to misread: loss absolute values by themselves do not carry a unified meaning. The discriminator loss 0.164 and the generator loss 1.609 use different scales and different directions—one is the average correctness cost for the two classes real and fake, the other is the cost of fake-sample realism. Moreover, these absolute values vary with the chosen loss variant (saturating or non-saturating, whether divided by 2, whether summed or averaged over the batch) and with the batch contents. Therefore, examining a single loss value alone cannot indicate whether sample quality is good or bad; it is only meaningful as a relative comparison under the same objective and scale. What really matters is the direction: a discriminator loss approaching 0 means nearly perfect discrimination, while a persistently high generator loss means fake samples have not yet fooled the adversary.

A complete small example can turn mode collapse from an abstract concept into a visible phenomenon. Suppose the real data consists of eight two-dimensional Gaussian clusters distributed on a circle, and the noise z is drawn from a two-dimensional Gaussian distribution. Alternately train the discriminator and the generator, and after each round plot the generated points together with the real clusters on the same figure for observation.

Early in training, the generator quickly finds a cluster to which the discriminator temporarily gives a high score, so it maps many different z values to the same region, and the outputs clump together. The discriminator then learns to recognize "the points in this cluster are too dense, and the distribution is wrong", and starts lowering the score of this region; the generator is forced to move, and jumps to another cluster to repeat the process. Throughout the process, the generated points always concentrate on a few clusters, the loss oscillates accordingly, but it never stably covers all eight clusters. This is the intuitive manifestation of mode collapse: the generator cheats by "only picking a few modes that are easy to fool the discriminator", rather than learning the complete real distribution.

The key question is: why can the discriminator still be temporarily fooled? Because every generated point the discriminator sees, taken individually, is sufficiently similar to points in the real clusters. If we only look at "how close this point is to the nearest cluster", it is entirely possible that all collapsed samples fall inside the real clusters, and quality metrics still look good. The discriminator can only detect the problem in a statistical sense—for example, that the density of some cluster is far higher than the real proportion—and discovering this requires enough samples and enough training steps, so the collapse can keep deceiving locally for a while.

Therefore, evaluation must change perspective. Do not just look at the distance from a single point to the nearest cluster; instead, count how many clusters the generated points cover, whether the proportion of each cluster is close to the real ratio, and whether neighborhoods of z map to nearby outputs (that is, points close in z-space should not scatter in output space; otherwise, the mapping is discontinuous). These metrics can reveal the problem that "every sample looks real, but the overall distribution is incomplete".

To address this collapse, several approaches can be tried: minibatch discrimination makes the discriminator simultaneously observe the correlations within a batch of samples, thereby detecting the shortcut of "the whole batch comes from the same mode"; spectral normalization constrains the Lipschitz constant of the discriminator and stabilizes its updates; switch to a Wasserstein objective, using a smoother distance signal to replace saturating JS-type objectives; or adjust model capacity and the data ratio. When comparing these modifications, use frozen checkpoints and the same number of samples as controls, and also check whether the improvement in sharpness comes at the cost of reduced coverage—sometimes new methods make individual samples more refined, at the cost of losing more clusters. The final point this example illustrates is that "every sample looks real" and "the generated distribution is complete" are two different things; the former only guarantees local realism, while the latter guarantees diversity.

Drawing GAN training as a loop diagram can directly answer a recurring question: why can neither a too-strong nor a too-weak discriminator teach the generator well? Figure 1 shows the complete structure of this dynamic game: noise z enters the generator and produces fake samples; fake samples and real samples are fed into the discriminator together; the discriminator gives real/fake judgments, and its gradient is then passed back to the generator, driving it to update. This "generator output → discriminator evaluation → gradient feedback → generator improvement" closed loop is the essence that distinguishes GAN from ordinary supervised learning—the loss is not fixed, but a signal given by the opponent in real time.

Figure 1 also depicts three states that this loop can fall into. When the capabilities and update cadence of both sides match, the loop tends toward equilibrium: the generator gradually approaches the real distribution, the discriminator maintains a boundary that is neither too strong nor too weak, and the two progress in sync. When the discriminator is too strong, the loop oscillates or even stagnates: the discriminator distinguishes real from fake almost perfectly, scores for fake samples are pushed close to zero, the gradient signal received by the generator approaches zero, and it learns no direction; and once the generator happens to find a breakthrough, the discriminator immediately adapts again, the two swing back and forth between extremes of strength and weakness, and the loss fluctuates up and down without converging. When the generator gets "stuck" in a local region, the loop manifests as mode collapse: the generator maps many different z to a few high-score regions, the discriminator temporarily cannot see through this laziness, and the two remain deadlocked on this crippled equilibrium.

All three states point to the same conclusion: GAN training is a dynamic game, and stability depends on the capabilities and update cadence of both sides. Capability imbalance allows one side to overwhelm the other, and update cadence imbalance causes the gradient signal and parameter updates to be misaligned. Understanding this loop, the subsequent discussions about loss selection, normalization, and update ratio are essentially all about finding operating conditions that can maintain balance for this loop.

If we temporarily hold the generator fixed and let only the discriminator optimize indefinitely, what will the ideal discriminator ultimately output? The answer to this question reveals what the GAN objective function is really measuring. For each input point x, the optimal discriminator converges to

D*(x) = p_data(x) / (p_data(x) + p_g(x))

Here, p_data(x) is the density of the real data at x, and p_g(x) is the density of the generated distribution at that point. This ratio has an intuitive meaning: at point x, the discriminator outputs the proportion of the total density that comes from the real data. If a point comes only from the real data (p_g = 0), D* gives 1; if it comes only from the generator (p_data = 0), D* gives 0; when the two densities are equal, D* = 1/2, and the discriminator admits it cannot distinguish them.

Substituting this optimal discriminator back into the original objective function turns the generator side into a quantity related to the JS divergence (Jensen–Shannon divergence) between the two distributions. That is, under the ideal assumption that the discriminator has reached optimality, the generator's minimization objective is equivalent to reducing the JS divergence between p_data and p_g, and the global minimum is achieved if and only if p_g = p_data, at which point D* = 1/2 for every point. This is the elegant theoretical closed loop: the solution to the minimax game is that the generated distribution exactly coincides with the real distribution.

But this conclusion relies on three assumptions that do not hold in real training: infinite discriminator capacity, infinite data, and the ability of both sides to fully converge to their respective optima in alternation. In actual training, D is a finite network, the data is a finite sample, and D and G are updated alternately, with neither ever truly converging. A more serious problem arises when the supports of the distributions barely overlap—when the real distribution and the generated distribution occupy almost disjoint regions, the JS divergence tends toward a constant, and its gradient saturates or even vanishes, leaving the generator without a useful directional signal. This exactly explains the motivation for later replacing the objective function and switching to other distribution distances (such as Wasserstein distance): it is not that the theory is wrong, but that the JS-type objective cannot provide a strong enough gradient to drive optimization in this common early situation of "non-overlapping support".

Why is −log D(G(z)) almost always used in practice instead of the original formula? The problem lies in the gradient curve of the saturating loss. The original generator objective is to minimize log(1 − D(G(z))). When the discriminator confidently classifies fake samples as fake, so D(G(z)) is close to 0, this term itself is already close to 0, and its gradient falls in the flat region of the sigmoid, so it barely changes with the parameters. At the start of training, the discriminator can often easily see through the generator, so the gradient the generator receives is almost zero, and learning freezes at the very beginning.

The non-saturating objective reverses the generator’s optimization direction: from minimizing log(1 − D(G(z))) to maximizing log D(G(z)), which is equivalent to minimizing −log D(G(z)). At the ideal equilibrium point it is completely consistent with the saturating form—both require p_g = p_data, and the discriminator is forced to output 1/2—so the optimal solution is unchanged. What changes is the path to that solution: when D(G(z)) is close to 0, the gradient of −log D(G(z)) is instead large, providing a strong signal to the generator early in training and pulling it out of the “completely seen through” state. This is exactly why the non-saturating loss is more commonly used: it fixes vanishing gradients, not the objective.

But its limits must also be clearly delineated. The non-saturating loss only solves one problem: “early gradients are too weak”; it does not fix mode coverage. The root causes of mode collapse are deeper: distribution matching itself may stop at a local solution when optimizing a JS-type objective, the dynamic game process allows the generator to slack off by relying on a few modes, and finite batches make it difficult for the discriminator to notice the overall lack of coverage—none of these causes can be eliminated by simply changing the generator loss term. Therefore the non-saturating loss is the default starting point for training GANs, but if samples begin to collapse and diversity decreases, one must turn to minibatch discrimination, spectral normalization, or Wasserstein objectives, rather than hoping to keep adjusting this term.

Why might a generator output 1000 clear, realistic faces while corresponding to only a few identity templates? The answer is that mode collapse is a diversity failure, not a quality failure. The generator can map many different noise z to the few modes currently most preferred by the discriminator: these modes are the regions where it can most easily fool the adversary, so it repeatedly generates "the few high-scoring faces", each one looking clear, realistic, and flawless when viewed individually, but together they reveal repetition—different inputs collapse to the same batch of outputs.

Complete collapse is easy to detect: all outputs are almost identical, and the problem is obvious at a glance. What is truly dangerous is partial collapse, which manifests as the absence of rare attributes, specific poses, or certain feature combinations. For example, a generator can draw faces of all age groups but almost never generate a certain rare angle, or a certain combination of skin tone and hairstyle; no single image looks problematic, and only when the whole distribution is examined do you notice that some regions have been systematically ignored. This kind of absence cannot be concluded from a sample grid—every image in the grid looks real.

Therefore, detecting mode collapse must use methods that reflect the "overall distribution" rather than spot-checking individual samples. You can compute nearest-neighbor distances between generated and real samples to see whether many generated samples crowd near a few real samples; you can interpolate in latent space and observe whether nearby z values map smoothly to nearby outputs or jump suddenly; you can count the coverage of classes or attributes to confirm that rare classes have not disappeared; you can use a precision–recall decomposition to separately measure "whether generated samples all look real" (precision) and "whether real samples are all covered" (recall); you can also use birthday-paradox-style duplicate detection—when diversity is insufficient, the probability of duplicate samples appearing is far higher than intuition suggests. Only such distribution-oriented checks can distinguish between "every sample looks real" and "the distribution is complete," which are two completely different things.

WGAN replaces the object measured by the entire objective: instead of having the discriminator output true/fake probabilities, it uses a real-valued critic to approximate the dual form of the Wasserstein-1 distance. The Wasserstein distance measures "the minimum cost required to transport one distribution into another," and its key advantage is continuity—even if the supports of the two distributions barely overlap, the Wasserstein distance still changes smoothly with the geometric distance between the two distributions, thus continuously providing meaningful gradient signals. This is precisely the shortcoming exposed earlier by JS divergence: when the supports do not overlap, JS divergence approaches a constant and suffers from vanishing gradients, while the Wasserstein distance does not.

For this dual form to hold, the critic must satisfy the 1-Lipschitz constraint, meaning that the rate of change of its output with respect to its input is restricted to a certain range; otherwise the dual formula is no longer equivalent to the true Wasserstein distance, and the critic can cheat by arbitrarily amplifying output differences. Early implementations used weight clipping to forcibly clamp each parameter within a fixed interval, which is simple but too crude—it distorts the critic's capacity boundary and can easily lead to gradient explosion or degradation. Later, gradient penalty (adding a penalty term to the loss for critic gradient norms deviating from 1) and spectral normalization (normalizing the spectral norm of each network layer) became more common alternatives, maintaining the Lipschitz constraint more smoothly.

With the critic, the loss value becomes more interpretable: it approximates a true distribution distance, and a decrease in value often means that the two distributions are really getting closer, rather than merely fooling some probability output. However, WGAN does not guarantee eliminating mode collapse—the continuous distance signal only makes optimization easier to carry out, and the collapse risk brought by distribution matching, game dynamics, and finite batches still exists. Moreover, both the strength of the Lipschitz constraint and the number of critic updates need to be validated in practice: if the constraint is too tight, it restricts the critic's expressive power, and if updates are too few, the critic cannot keep up with the generator; both cause the dual approximation to fail.

When the discriminator accuracy is close to 100%, should we continue training it to be stronger? Usually not. Discriminator accuracy approaching 100% often means it has already overfit the limited data—it no longer outputs a generalization signal about "the difference between real and fake distributions", but instead memorizes the training samples themselves. At this point, the gradients it gives to the generator no longer point to "how to get closer to the real distribution", but to "how to reproduce that batch of samples it has memorized", which is almost useless or even harmful to the generator. Conversely, a discriminator that is too weak also won't work: it can't distinguish the difference, gives the generator ambiguous directions, and the two networks will wander together at a low level.

Therefore, training stability first lies in the "update ratio": adjusting the learning rates of the discriminator and generator, as well as how many times each is updated per round. The discriminator usually needs more updates to maintain an appropriate decision boundary, but not so many that it overfits. At the same time, spectral normalization can constrain the Lipschitz constant of the discriminator, preventing its output from being too sensitive to inputs and amplifying noise; regularization suppresses overfitting; data augmentation expands the effective sample size; batch statistics (such as the mean and variance of batch normalization) affect the stability of gradients across batches. The goal of all these measures is to roughly match the capabilities and update rhythms of D and G, that is, to keep the feedback loop discussed in the previous chapter in the equilibrium zone.

When monitoring training, observe several signals rather than only looking at a single loss: whether the gradient norm explodes or vanishes, whether the generator's output distribution is reasonable, and whether sample coverage is sufficient. Also pay attention to two common misjudgments. Oscillation of the training loss does not necessarily mean failure—the two adversaries taking turns leading will naturally make the loss fluctuate; conversely, a smooth decline in loss is also not a guarantee of success, because mode collapse can quietly occur while the loss appears stable. The truly reliable basis is to save checkpoints regularly and use external evaluations decoupled from training (such as the coverage, memorization, and stability metrics discussed later) to judge the actual quality of each checkpoint, rather than staring at the training curve and drawing conclusions.

GAN's generator is a direct mapping from z to x: given a noise vector, one forward pass produces a complete sample, without step-by-step iteration. This makes it especially suited to low-latency sharp generation—in scenarios that require fast image output, this speed advantage is very prominent.

But the direct mapping is both a strength and a weakness. Given a real image, how do you find the corresponding z and edit it? The generator typically has no closed-form inverse mapping; that is, there is no ready-made function that inverts x back into z. To perform inversion, you can only optimize a z in latent space so that G(z) is as close as possible to the target image, or additionally train an encoder to approximate the inverse mapping. Either way, inversion introduces errors, and these errors alter the identity and details of the original image—the edited result may no longer be faithful to the original, and a person's features and object textures will drift during the round trip.

Likelihood is similarly limited. GAN learns an implicit distribution—it can sample, but cannot give the exact probability density of any sample point. This means computing p(x) is difficult, and tasks that rely on explicit likelihood, such as anomaly detection and coverage proofs, are therefore harder to apply directly. You cannot simply read out “what is the probability of this image appearing?” as you can in autoregressive or flow models.

These limitations in turn define the applicable boundaries of GANs. It is suited to generation tasks that prioritize speed and sharpness; while for tasks requiring precise coverage, explicit control, invertible encoding, or strict likelihood computation, diffusion models, flow models, and autoregressive models each offer different trade-offs—they often sacrifice single-step generation speed in exchange for more stable training, more complete distribution coverage, or stronger controllability. Choosing which generative framework to use is essentially a trade-off among speed, likelihood, coverage, and invertibility.

A common misconception is: if FID is lower, does that rule out the possibility that training samples have been copied? No. FID works by passing real samples and generated samples separately through a pre-trained feature network and comparing the first-order means and second-order covariances of the two sets of features. It is good at capturing overall distribution shift, but it may miss sparse modes—a rare class that disappears entirely may contribute so little to the overall mean and covariance that it is ignored—and it is also susceptible to bias from small-sample estimates. Moreover, FID does not answer the memorization question at all: the generator can memorize and output training samples one by one, and FID will still be very low.

Therefore, evaluation must consider at least four dimensions at the same time: quality, coverage, memorization, and training stability. Quality and coverage can be measured separately using the precision–recall decomposition—precision measures whether all generated samples are realistic (i.e., whether they look real), and recall measures whether all real samples are covered (i.e., whether coverage is complete). For conditional generation tasks, class accuracy and human judgment must also be added to confirm that the condition is actually satisfied. Memorization must be checked using training-set nearest-neighbor retrieval, membership inference, and duplication rate: compute the nearest-neighbor distances between generated samples and training samples, look for signs of sample-by-sample copying, and detect abnormally high duplication rates.

Training stability is the fourth dimension. It requires fixing the number of samples, the feature network, and the preprocessing method when reporting results; otherwise metrics such as FID are not comparable. More importantly, report results from multiple random seeds and the proportion of failed training runs—showing only successful runs systematically overestimates stability, because the same configuration may collapse or diverge under most seeds, while the selected "best-looking image" hides all of this. Only by reporting the failure rate together with the mean and variance will the evaluation conclusion be credible.

String all the previous steps into a causal chain, and you can see GAN move from "an idea" step by step to "verifiable practice"; the output of each step is the input to the next.

The starting point is noise z, which passes through generator G to produce an implicit distribution—it has no explicit probability formula and can only be expressed through sampling. Next, discriminator D appears, comparing real samples with generated samples and learning the current difference between the two distributions. This difference is then backpropagated through gradients to the generator, which updates accordingly to make fake samples more like real ones. So far a closed loop is formed, but the loop is not automatically stable: you must balance the capabilities and update rhythms of both sides; otherwise one side overwhelms the other, and gradients either vanish or explode. Therefore, during training you must continuously monitor two signals—mode collapse and gradient anomalies—which are the first failure points detectable on this chain.

When a failure appears, improvement measures trace back along the causal chain: changing the objective function (such as Wasserstein distance), applying spectral normalization or regularization are all enhancing the stability of this feedback loop. But "training looks stable" is not enough; ultimately evaluation must be brought back to four dimensions and verified simultaneously: realism, coverage, memorization, and failure rate—none can be omitted.

This chain from principle to practice also corresponds to a set of verifiable validation layers. To verify whether a change is truly effective, first fix the input: use the same batch of samples, the same preprocessing and permission boundaries, record input hashes, slice labels, and rejection reasons to ensure the starting point of comparison is consistent. Second, lock the mechanism: change only one core variable, lock all other configurations, then observe key intermediate states and the position where the first deviation from expectation occurs—only then can the effect be attributed to that variable. Third, unify the output: adopt the same acceptance rules and resource budget, and measure layered differences in quality, cost, latency, and failure rate. Finally, retain the counter-evidence: there must be a control group that does not enable the target mechanism, to confirm whether the benefit can be stably replicated across samples and random seeds, rather than being an illusion of a single lucky run.

Looking at the entire chain of "Generative Adversarial Network (GAN): Letting the Generator Learn the Data Distribution Against a Dynamic Discriminator", every claim can land on an observable piece of evidence: distribution difference is seen in gradients and loss, stability in failure rate and oscillation, diversity in coverage and duplication rate, authenticity in precision and human judgment, memory in training set nearest neighbors. Every link of the causal chain is ultimately closed by some measurable output.