Practical Lessons for Training GANs

1. Start Simple, Then Scale

Don't jump straight to your full dataset at full resolution. Build confidence in your pipeline first:

Overfit on one image first. If your GAN can't memorize a single image, it definitely can't learn a distribution. This validates your architecture and training loop.
Then try 10 images. Can it learn a tiny distribution? Good. Now you know the adversarial dynamics work.
Start at low resolution. Train at 64x64 before attempting 256x256 or higher. Debug faster, iterate faster.
Use a known dataset first. Before training on your custom data, verify your code works on MNIST or CelebA. If it doesn't work on a standard benchmark, the problem is your code, not your data.

2. Learning Rates Are Everything

The balance between generator and discriminator learning rates is the single most impactful hyperparameter:

📉

Use a Lower Generator LR

A common starting point: discriminator at 1e-4, generator at 1e-4 or slightly lower. If the generator learns too fast, it can find degenerate shortcuts. If the discriminator runs away, the generator gets no useful gradient signal.

🧪

TTUR (Two Time-Scale Update Rule)

Use separate learning rates for G and D. TTUR (Heusel et al., 2017) proved that GANs trained with two time-scale updates converge to a local Nash equilibrium. The widely used setting of 4e-4 for D and 1e-4 for G was popularized by SAGAN (Zhang et al., 2019): a faster discriminator supplies more informative gradients to the generator.

⚠️

Adam Betas Matter

The default Adam beta1 of 0.9 is often too high for GANs. DCGAN (Radford et al., 2016) used beta1=0.5; WGAN-GP (Gulrajani et al., 2017) used beta1=0.0 with beta2=0.9. High momentum in Adam can cause oscillations in the adversarial game.

3. Fighting Mode Collapse

Mode collapse is when the generator finds a small set of outputs that fool the discriminator and stops exploring. The generator covers only a few modes of the data distribution. Signs to watch for:

Red Flags

Generated images all look nearly identical
Loss oscillates wildly, or (with standard GAN losses) discriminator loss drops to zero
Generator loss keeps decreasing but image quality stagnates
FID score plateaus while loss still improves

Established countermeasures:

Minibatch discrimination: Let the discriminator see statistics across the batch, not just individual samples. If all samples look the same, the discriminator can detect it (Salimans et al., 2016).
Feature matching: Instead of training G to maximize D's error, train it to match the statistics of real data in D's intermediate layers (Salimans et al., 2016).
Use Wasserstein loss: WGAN (Arjovsky et al., 2017) and WGAN-GP (Gulrajani et al., 2017) provide more stable gradients and a loss that tracks sample quality. The Earth Mover's distance does not saturate like the original GAN loss.
Multiple critic steps per generator step (n_critic): WGAN and WGAN-GP update the critic several times (typically 5) per generator update, so the generator always receives gradients from a near-optimal critic.
Unrolled GANs: Backpropagate the generator update through k simulated discriminator updates, so the generator is optimized against how D will respond rather than D's current state, discouraging it from collapsing onto a single mode (Metz et al., 2017).

4. Architecture Choices That Matter

Avoid

Max pooling (loses spatial information)
BatchNorm in D when using WGAN-GP: the per-sample gradient penalty is invalid when samples in a batch interact (DCGAN itself used BatchNorm in D)
Sigmoid output in D with WGAN loss
Sparse gradients in D (ReLU, max pooling); prefer LeakyReLU
Fully connected layers in deep architectures
Very deep networks without residual connections

Spectral Normalization

One of the most impactful stabilization techniques (Miyato et al., 2018). It normalizes the weight matrices of the discriminator by their spectral norm (largest singular value), enforcing a Lipschitz constraint without the computational overhead of gradient penalty. In many settings, SpectralNorm in D is sufficient without a gradient penalty.

5. Data Preparation Is Half the Battle

Data quality matters even more for GANs than for supervised models:

Clean your dataset aggressively. A few corrupted, mislabeled, or drastically out-of-distribution images can derail training. The discriminator will latch onto artifacts.
Align and crop consistently. If you are generating faces, align them. If you are generating snow crystals, center them. Spatial consistency helps the generator focus on content rather than position.
Augment carefully. Random flips and small rotations are usually safe. Color jitter and aggressive crops can confuse the distribution the generator is trying to learn.
More data beats more training. If results are poor, consider collecting more data before training longer. Discriminators overfit small datasets quickly.
Normalize to [-1, 1]. Match the tanh output range. This sounds obvious but is a common source of bugs.

6. Monitoring Training

GAN loss curves are often uninformative. Do not rely on them alone:

🖼️

Save Generated Samples Frequently

Generate and save a grid of images every N steps using a fixed latent vector. This lets you visually track improvement and catch mode collapse early. Visual inspection is often more informative than the loss curve.

📊

Track FID Score

Fréchet Inception Distance (FID; Heusel et al., 2017) compares the statistics of generated images to real images in Inception feature space. Lower is better. Compute it periodically rather than every step, since it is expensive. It is the most widely used quantitative quality metric, though it is less reliable on small datasets and on imagery far from Inception's training domain.

⚖️

Watch the D/G Loss Balance

For WGAN and WGAN-GP, the critic loss is approximately the negative of the critic's Wasserstein estimate, so its magnitude should shrink gradually as the generator improves. A value near zero means the critic can no longer separate real from fake: either the generator matches the data, or the critic is too weak (check the samples to tell which). A rapidly growing magnitude means the critic dominates. Large oscillations usually indicate learning rates that are too high. For the standard (BCE) GAN loss, D's loss per real/fake term sits near ln 2 ≈ 0.69 at equilibrium; D loss persistently near zero means D is overpowering G.

7. Choosing a Loss Function

The loss function shapes the entire training dynamic. A practical comparison:

Loss	Stability	Quality	Notes
Vanilla (BCE)	Low	Moderate	Prone to vanishing gradients and mode collapse
WGAN	High	Good	Requires weight clipping, which can limit capacity
WGAN-GP	High	High	Gradient penalty is more principled than clipping
Hinge	High	High	Used in SAGAN (Zhang et al., 2019) and BigGAN (Brock et al., 2019). Simple and effective
Non-saturating	Moderate	Good	-log(D(G(z))) instead of log(1-D(G(z))). Better gradients early on

Recommendation: Start with WGAN-GP or hinge loss; both train stably in practice. Only the WGAN family yields a loss curve that estimates a distance between distributions and tracks sample quality; hinge loss curves are not a distance estimate. Use the original GAN loss only with a specific reason.

8. Working with Small Datasets

Not everyone has millions of images. When your dataset is small (hundreds to low thousands):

📐

Progressive Growing

Start at low resolution and scale up (Karras et al., 2018). Early phases learn coarse structure efficiently.

🔄

Data Augmentation

Apply differentiable augmentation to both real and fake images (DiffAugment; Zhao et al., 2020). This reduces discriminator overfitting.

🧊

Freeze D Layers

When fine-tuning a pretrained GAN on limited data, freezing the discriminator's early layers reduces overfitting (FreezeD; Mo et al., 2020).

The snowGAN project was trained on the 2,341-image 2024–2025 release of the Rocky Mountain Snowpack dataset using progressive growing: a small, domain-specific dataset where these techniques matter most.

9. Common Pitfalls & Quick Fixes

Checkerboard artifacts in generated images

Caused by uneven overlap in transposed convolutions, particularly when the kernel size is not divisible by the stride (Odena et al., 2016). Fix: use nearest-neighbor upsampling followed by a regular convolution instead of ConvTranspose2d.

Generator produces noise / no recognizable structure

The discriminator is likely too strong. Lower D's learning rate, reduce D updates per G update, or use one-sided label smoothing, or instance noise on D's inputs.

Training loss oscillates wildly

Learning rates are too high, or batch size is too small. Try halving both LRs first. If that does not help, increase batch size or add gradient penalty.

Good samples early, then quality degrades

The generator may be overfitting or the adversarial balance shifted. Save checkpoints frequently and consider learning rate decay in late training.

NaN losses

Usually gradient explosion. Add gradient clipping (max_norm=1.0), lower learning rates, or check for division by zero in custom loss functions. Also verify that your data does not contain NaN/inf values.

The GAN Training Checklist

Before training, confirm:

☐ Data cleaned, aligned, and normalized to [-1, 1]
☐ Architecture validated on a known dataset
☐ Single-image overfit test passed
☐ Learning rates set (try 1e-4 / 1e-4 or TTUR)
☐ Adam betas adjusted (beta1=0.0 or 0.5)
☐ Loss function chosen (WGAN-GP or hinge recommended)
☐ Image saving callback configured (every N steps)
☐ Checkpoint saving enabled
☐ FID evaluation scheduled
☐ GPU memory checked (reduce batch size if needed)

References

Radford, A., Metz, L., & Chintala, S. (2016). Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR. arxiv.org/abs/1511.06434
Salimans, T., et al. (2016). Improved techniques for training GANs. NeurIPS. arxiv.org/abs/1606.03498
Odena, A., Dumoulin, V., & Olah, C. (2016). Deconvolution and checkerboard artifacts. Distill. distill.pub/2016/deconv-checkerboard
Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. ICML. arxiv.org/abs/1701.07875
Gulrajani, I., et al. (2017). Improved training of Wasserstein GANs. NeurIPS. arxiv.org/abs/1704.00028
Heusel, M., et al. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. NeurIPS. arxiv.org/abs/1706.08500
Metz, L., Poole, B., Pfau, D., & Sohl-Dickstein, J. (2017). Unrolled generative adversarial networks. ICLR. arxiv.org/abs/1611.02163
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2018). Progressive growing of GANs for improved quality, stability, and variation. ICLR. arxiv.org/abs/1710.10196
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. ICLR. arxiv.org/abs/1802.05957
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2019). Self-attention generative adversarial networks. ICML. arxiv.org/abs/1805.08318
Brock, A., Donahue, J., & Simonyan, K. (2019). Large scale GAN training for high fidelity natural image synthesis. ICLR. arxiv.org/abs/1809.11096
Mo, S., Cho, M., & Shin, J. (2020). Freeze the discriminator: A simple baseline for fine-tuning GANs. arXiv. arxiv.org/abs/2002.10964
Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y., & Han, S. (2020). Differentiable augmentation for data-efficient GAN training. NeurIPS. arxiv.org/abs/2006.10738