Progressive Growing of GANs

Why Training Big GANs Is Hard

Training a GAN to generate 1024×1024 images directly is difficult for several reasons:

Unstable gradients: The generator must simultaneously learn coarse structure and fine detail across a massive parameter space. The optimization problem is poorly conditioned.
Discriminator dominance: At high resolution, even tiny imperfections are easy to spot. The discriminator overwhelms the generator before it learns anything meaningful.
Computational waste: Early training, when the network is still learning coarse shapes, is spent on expensive full-resolution computation.
Mode collapse: The simultaneous learning of all scales increases the likelihood the generator collapses to producing narrow, repetitive outputs.

Progressive growing mitigates these issues by decomposing the problem into a series of simpler sub-problems.

How Progressive Growing Works

The Original Paper

Progressive Growing of GANs for Improved Quality, Stability, and Variation by Karras, Aila, Laine, and Lehtinen (ICLR 2018; arXiv:1710.10196). The paper demonstrated high-quality 1024×1024 face synthesis on CelebA-HQ and began the lineage that led to StyleGAN and StyleGAN2.

The training progression follows a resolution ladder. Both the generator and discriminator grow in lockstep:

Phase 1

4x4

→

Phase 2

8x8

→

Phase 3

16x16

→

Phase 4

32x32

→

...

1024

At each phase, new layers are added to both networks. The generator grows at its output end (producing higher-resolution images), while the discriminator grows at its input end (accepting higher-resolution images). Existing layers keep their learned weights and remain trainable.

Phase	Resolution	What the Network Learns
1	4x4	Overall color, basic blob structure
2-3	8x8 - 16x16	Rough spatial layout, coarse shapes
4-5	32x32 - 64x64	Recognizable features, basic textures
6-7	128x128 - 256x256	Fine details, complex textures
8-9	512x512 - 1024x1024	Highest-fidelity micro-details

The Fade-In: Smooth Transitions

Adding a new layer abruptly would disrupt training: its randomly initialized weights would degrade what the network has already learned. The solution: fade in new layers smoothly using a blending parameter α that transitions from 0 to 1.

output = (1 - α) × upsampled_old + α × new_layer

Alpha Fade-In Progress

α = 0.0

Old only

α = 0.25

Blending

α = 0.5

Half & half

α = 0.75

Blending

α = 1.0

New only

When α = 0, the output comes entirely from the upsampled previous resolution. As training progresses, α linearly increases to 1, at which point the new layer has fully taken over. The old shortcut path is then removed. Each resolution phase consists of a transition period (fade-in) followed by a stabilization period (training at the new resolution with α fixed at 1.0).

Supporting Techniques

The ProGAN paper (Karras et al., 2018) pairs progressive growing with several auxiliary techniques that keep training stable:

⚖️

Equalized Learning Rate

Instead of careful weight initialization, all weights start from N(0,1) and are scaled at runtime by the He constant. This ensures all parameters have the same dynamic range regardless of layer depth, preventing some layers from learning faster than others.

📏

Pixelwise Feature Normalization

Applied after each generator convolution, this normalizes the feature vector at each pixel to unit length. This prevents signal magnitudes from escalating during the adversarial feedback loop.

📊

Minibatch Standard Deviation

A layer near the end of the discriminator that computes the statistical variation across the minibatch and concatenates it as an extra feature channel. If the generator produces low-diversity outputs, this statistical signature gives it away.

📐

WGAN-GP Loss

Wasserstein loss with gradient penalty (Gulrajani et al., 2017) provides stable training and a loss that tracks sample quality. The gradient penalty enforces a Lipschitz constraint by penalizing gradient norms that deviate from 1.

Case Study: The Abominable SnowGAN

snowGAN

A progressive GAN built to generate synthetic images of magnified snowpack. It is trained on winter snowpack photography collected in the Colorado Rocky Mountains and outputs 1024×1024 RGB images of snow crystal formations.

The snowGAN project applies progressive growing to an unusual domain: snow crystal imagery. The implementation departs from the original recipe in two ways: the generator starts at 16×16 rather than 4×4, and it upsamples with transposed convolutions rather than nearest-neighbor upsampling followed by convolution, which carries the checkerboard-artifact risk described in Practical Lessons for Training GANs. The architecture is as follows:

Architecture

Generator

Input: 100-dim latent vector
Initial projection to 16x16 feature map
Transposed convolution stack with configurable filters
Fade-in blocks for smooth resolution transitions
Final activation: tanh (output range [-1, 1])
Output: 1024x1024 RGB images

Discriminator

Input: 1024x1024 RGB images
Conv2D blocks with LeakyReLU
Wasserstein output (no activation)
Gradient penalty for stability
Optional dual-head inference mode

The Fade-In Implementation

snowGAN's fade-in implementation splits the generator output into two paths during resolution transitions:

Generator during fade-in (e.g., 16x16 → 32x32): ┌─────────────────────────┐ │ Existing 16x16 block │ └────────────┬────────────┘ │ ┌────────┴────────┐ │ │ ┌───────▼───────┐ ┌──────▼──────┐ │ Upsample to │ │ New 32x32 │ │ 32x32 + │ │ conv block │ │ old toRGB │ │ + new toRGB │ └───────┬───────┘ └──────┬──────┘ │ │ ▼ ▼ (1 - alpha) * alpha * prev_output curr_output │ │ └────────┬────────┘ ▼ Blended output

The blending alpha is computed from the current training step:

alpha = clamp(current_step / total_fade_steps, 0.0, 1.0)

Training Data

snowGAN trains on the Rocky Mountain Snowpack Dataset (2,341 samples from the 2024–2025 Colorado winter season). Each sample includes magnified crystal images, snowpack profiles, and core segments, along with metadata like GPS coordinates, temperatures, slope angles, and avalanche observations.

With 2,341 samples, this is a small dataset for GAN training. Progressive growing is well suited to it because the early low-resolution phases can learn coarse structure from limited data, where a GAN trained directly at full resolution would be more likely to struggle.

Beyond Generation: Dual-Head Inference

snowGAN also repurposes its trained discriminator as a feature extractor. The discriminator's output layer is stripped and replaced with two classification heads:

Avalanche Count

21 classes predicting the number of avalanches observed at the sampling site. Leverages the discriminator's learned snow crystal representations.

Wind Loading

4 classes (high/medium/low/none) predicting wind loading conditions. The aim is for the discriminator's learned crystal-structure features to transfer to this safety-relevant task.

This is an application of transfer learning: adversarial training pushes the discriminator to develop rich feature representations of snow crystals, which can then be reused for downstream tasks instead of training a new model from scratch.

Why Progressive Growing Matters

⚡

2-6x Faster

Early phases on small images are cheap; the ProGAN authors report a 2–6× training speedup depending on final resolution.

🎯

Higher Quality

ProGAN demonstrated high-quality 1024×1024 face synthesis. Learning structure before detail improves results.

🔧

More Stable

Each stage is an easier sub-problem, which avoids abrupt resolution jumps and reduces mode collapse.

The broader lesson extends well beyond GANs: curriculum learning works. Start simple, increase complexity gradually, and preserve what has already been learned. Progressive growing demonstrated this for GAN image synthesis, and coarse-to-fine training strategies appear across generative modeling.

The Legacy: ProGAN to StyleGAN

Progressive growing also began a line of influential GAN architectures:

ProGAN (2018): The original progressive growing paper (Karras et al., 2018).
StyleGAN (2019): Built on ProGAN's architecture and training setup; introduced a style-based generator using AdaIN and a more disentangled intermediate latent space (Karras et al., 2019).
StyleGAN2 (2020): Removed progressive growing in favor of other architectural changes, while retaining the coarse-to-fine design philosophy (Karras et al., 2020).
StyleGAN3 (2021): Addressed aliasing to make generation equivariant to translation and rotation (Karras et al., 2021).

This ProGAN → StyleGAN lineage shaped subsequent GAN-based image synthesis research.

References

Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2018). Progressive growing of GANs for improved quality, stability, and variation. ICLR. arxiv.org/abs/1710.10196
Gulrajani, I., et al. (2017). Improved training of Wasserstein GANs. NeurIPS. arxiv.org/abs/1704.00028
Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. CVPR. arxiv.org/abs/1812.04948
Karras, T., et al. (2020). Analyzing and improving the image quality of StyleGAN. CVPR. arxiv.org/abs/1912.04958
Karras, T., et al. (2021). Alias-free generative adversarial networks. NeurIPS. arxiv.org/abs/2106.12423

The Core Idea