Diffusion models have surpassed GANs in image quality and diversity, and they train more stably. GANs are faster at sampling, because diffusion models need multiple denoising steps, each a forward pass. Hybrids such as denoising diffusion GANs and adversarially distilled diffusion models try to give you the coverage of diffusion at close to GAN speed. Which one suits a given system comes down to three things: how many milliseconds you can spend per frame, how much of the face and expression space the output has to cover, and how much training data and compute you have.
Two ways to turn noise into a face
A GAN trains two networks against each other. The generator turns Gaussian noise into a sample, and the discriminator estimates the probability that a sample is real rather than generated. Goodfellow et al. (2014) set training up as a minimax two-player game, in which the generator tries to maximise the chance that the discriminator makes a mistake. After training, generating an image means running the generator once.
A diffusion model uses two processes instead. The forward process adds Gaussian noise to a training image over T steps until it is close to pure noise. The reverse process, which the model learns, removes that noise step by step to produce a new image. In the usual formulation, training comes down to minimising a weighted mean squared error between the noise that was actually added and the noise the network predicts.
GAN methods learn from an adversarial loss and diffusion methods from a denoising loss. The first is a two-player minimax game between generator and discriminator. The second minimises a mean squared error on the added noise.
Why do diffusion models train more stably than GANs?
Dhariwal and Nichol (2021) count a stationary training objective among the properties that make diffusion models attractive. A GAN survey says the discriminator forgets earlier generated samples, and because each loss depends on the other network’s performance, this forgetting causes instability.
Early in training, generated samples are poor and the discriminator separates real from fake easily. Its gradient then vanishes and the generator stops updating. Simultaneous gradient descent, the standard way to train both networks, often fails to find even a local Nash equilibrium. Mescheder et al. (2018) showed that unregularised GAN training does not always converge in the more realistic case, where the distributions are not absolutely continuous. The result is parameters that oscillate and never converge, along with mode collapse and diverging behaviour.
The authors of the denoising diffusion GAN paper give two possible reasons. One is the difficulty of generating a sample from a complex distribution in one shot; the other is overfitting when the discriminator only looks at clean samples. Their hybrid splits generation into conditional denoising steps. Each step is simpler to model, and the diffusion process smooths the distributions the discriminator compares. They report stable training with no loss explosions. They put this down to strong conditioning on the noisy input, which keeps generator and discriminator in balance, and to training on smoother distributions.
Diffusion training is not automatically stable either. Nichol and Dhariwal found the theoretically motivated variational bound objective hard to optimise in practice on ImageNet 64×64, and preferred a hybrid objective. The simple loss proposed by Ho et al. turned out to be stable to optimise. In our reading, the stability comes from that choice of objective, not from diffusion as a category.
Modern GANs push back on the “hard to train” reputation
The R3GAN paper disputes the widely-spread claim that GANs are hard to train. It derives a regularised relativistic loss that deals with mode dropping and non-convergence without the usual collection of ad hoc tricks. Its authors report that the resulting minimal baseline beats StyleGAN2 on FFHQ, ImageNet, CIFAR and Stacked MNIST, and compares favourably with diffusion models. Earlier, Mescheder et al. had shown that simplified gradient penalties give local convergence and can train high-resolution image models with little hyperparameter tuning.
We read the instability folklore as an accurate description of unregularised GANs and a much weaker description of well-regularised modern ones. Scaling up is still delicate: the GigaGAN authors found that simply increasing StyleGAN’s capacity quickly becomes unstable.
Why GANs generate faster than diffusion models at inference
A GAN makes an image in one shot, with a single forward pass. A diffusion model needs one forward pass per denoising step. The step count is large because of a modelling assumption. Diffusion models usually assume each reverse step is Gaussian, and that only holds for very small steps, so the reverse process needs hundreds to thousands of them. Nichol and Dhariwal trained with 4,000 diffusion steps, which meant a single sample took several minutes on a modern GPU. GigaGAN, by contrast, produces a 512px image in 0.13 seconds.
Four lines of work have narrowed the gap:
- Fewer steps from the same model. DDIM (Song, Meng and Ermon) samples 10× to 50× faster in wall-clock time than DDPM, using the same training procedure. Learned variances let improved DDPMs produce good samples in as few as 50 forward passes.
- Distillation. Salimans and Ho distilled samplers from as many as 8,192 steps down to 4, reaching an FID of 3.0 on CIFAR-10. Consistency models generate in one step, with an FID of 3.55 on CIFAR-10.
- Adversarial distillation. Adversarial Diffusion Distillation (Sauer et al., Stability AI) samples large diffusion models in 1 to 4 steps. Its authors report beating few-step GANs in a single step and matching SDXL in four.
- Latent space. Latent diffusion runs denoising in the latent space of a pretrained autoencoder, which needs much less compute than pixel-space diffusion.
Hybrids go further. Denoising diffusion GANs use a conditional GAN to model each large, multimodal denoising step, so they can generate in as few as two steps. On CIFAR-10 that is about 2,000× faster than predictor-corrector sampling. In image editing at 256px, it meant 0.16 seconds per image against 181 seconds for the method they compared with. A latent variant, LDDGAN, reports sampling 1,000 times faster than DDPM with comparable FID. Among the GANs it was compared with, only StyleGAN2 with adaptive data augmentation had slightly better sample quality.
GigaGAN was at least 20 times faster than the diffusion models it was tested against, and stayed faster than progressively distilled Stable Diffusion. Like the Adversarial Diffusion Distillation result above, this is a benchmark reported by the method’s own authors. Treat it as a direction, not a number that will hold at your resolution and on your hardware.
Diversity and mode collapse: what FID hides
GANs tend to give up diversity in exchange for sharpness, and precision and recall show this more clearly than FID does. In mode collapse, the generator produces samples from a few modes instead of the whole data distribution. A GAN trained on handwritten digits, which have ten modes, may never produce some of the digits. For a face generator, the equivalent would be convincing faces from only a narrow range of ages, poses or skin tones.
FID alone will not tell you which failure is happening. Sajjadi et al. (2018) showed that FID responds to mode dropping but reduces everything to one number. It cannot tell mode dropping apart from mode inventing, or tell a model that renders a few digits well from one that renders all digits poorly. Precision and recall can. Their measurements confirmed the folklore that GANs often produce sharp images but suffer from mode collapse: high precision, low recall.
Diffusion models measured with precision and recall cover more of the distribution. Nichol and Dhariwal found that BigGAN-deep beat their smaller diffusion model on FID but did worse on recall, and that diffusion models reach much higher recall at similar FID. Dhariwal and Nichol (2021) then reported diffusion models with perceptual quality similar to GANs and higher recall. You can see this in samples that include modes the GAN missed.
Two caveats stop this from being a clean win. First, both families have a dial that trades diversity for fidelity: truncation for GANs and classifier guidance for diffusion. The GigaGAN authors note that truncation’s effect is quantitatively similar to guidance. Pushing classifier guidance beyond 1.0 smoothly trades recall for precision. Second, diffusion models can collapse too. In an 8-shot SAR study, EDM2 produced realistic images but suffered severe mode collapse, copying training samples with almost no variation.
How much more compute does a diffusion model need than a comparable GAN?
For training, not necessarily more in Dhariwal and Nichol’s comparison. For inference, more, since each sampling step is a forward pass.
On training, Dhariwal and Nichol (2021) compared their models’ training compute in V100-days with StyleGAN2 and BigGAN-deep, and reported better FID for a similar compute budget. Their ImageNet 128×128 model passed BigGAN-deep’s FID of 6.02 after 500K iterations, one eighth of the way through training. Where the model runs matters: Rombach et al. note that training powerful pixel-space diffusion models often takes hundreds of GPU days, and latent diffusion was designed to cut that cost.
On inference, the arithmetic is simple. A diffusion model needs several forward passes per sample where a GAN needs one. That makes step reduction the first lever to pull when serving GANs and diffusion models in production.
Published training times for hybrids give a rough sense of scale for the middle ground. Denoising diffusion GANs took about 48 hours on 4 V100 GPUs for CIFAR-10, and 180 hours on 8 V100s for CelebA-HQ. The latent variant reports 1.5 days on 2 GPUs for CelebA-HQ. These figures are for hybrids, so they do not settle the pure GAN against pure diffusion question.
GAN vs diffusion for deepfake generation: face swap, reenactment and video
A 2024 survey of deepfake generation and detection finds that GANs and diffusion models are the most common approaches to deepfake generation, with transformer-based methods also popular for video and audio. GAN-based deepfakes have mostly targeted the face: synthesising faces of people who do not exist, swapping faces, and editing attributes such as age or expression.
GAN pipelines handle face swap and reenactment. FSGAN and FSGAN2 provide a two-stage pipeline that handles both face swapping and reenactment. Identity-injection designs such as SimSwap came later, but these methods are still limited by low resolution, loss of facial attributes and poor handling of occlusion. One survey blames the resolution ceiling on unstable GAN training. Facial attribute editing still relies mainly on GAN-based disentanglement, and diffusion models are only gradually entering that area.
Diffusion models are used for text-driven and identity-personalised generation. With text-to-image models, users can produce content featuring public figures just by putting their names in the prompt. Stable Diffusion, one of the most popular of these models, runs diffusion in the latent space of a VQGAN. Test-time fine-tuning methods preserve identity well but take a long time to generate. Video is harder for diffusion: the main challenge is adding conditioning while keeping frames consistent over time. The output often needs a super-resolution stage later in the pipeline. In talking-head generation, Diffusion Heads uses an autoregressive diffusion model driven by a single identity image and an audio sequence.
The two families are also combined inside single systems. One line of work represents identity in Stable Diffusion using StyleGAN’s latent space, fed into the U-Net through cross-attention layers.
Detectors trained on GAN fakes miss diffusion fakes
The two families leave different traces, which matters to anyone building or evaluating a deepfake detector. GAN images carry a periodic, grid-like pattern in their frequency spectrum, caused by transposed convolutions. Diffusion images mostly lack that grid but tend to underestimate high frequencies. Ricker et al. trace this to the training objective, which gives less weight to the low-noise steps that control fine detail. Latent diffusion is the exception. Its images show a visible grid in the spectrum, possibly because the final image comes from an adversarially trained autoencoder.
The practical result is that detection generalises in one direction only. In Ricker et al.’s tests on diffusion-generated images, state-of-the-art GAN detectors lost 15.2% AUROC on average. Retrained on diffusion images, the same detectors reached almost perfect detection, and that carried over to GANs. Training on GAN images did not carry over to diffusion images. A detector trained only on GAN-era datasets is tuned to an artefact that newer generators mostly do not produce, so its benchmark scores overstate how well it protects you today.
When to choose a GAN, a diffusion model or a hybrid
| If the system needs | Lean towards | Reason |
|---|---|---|
| Real-time or interactive generation per frame | GAN, or a distilled or hybrid diffusion model | A GAN needs one forward pass; GigaGAN produces a 512px image in 0.13 seconds |
| Broad coverage of identities, poses and expressions | Diffusion | Diffusion models reach much higher recall than GANs at similar FID |
| Text prompts or other flexible conditioning | Latent diffusion | Cross-attention layers accept text or bounding boxes as conditioning |
| Latent interpolation and style mixing for editing | GAN | GigaGAN keeps a disentangled latent space that allows coarse and fine styles to be blended |
| A super-resolution stage behind another generator | GAN upsampler | GigaGAN’s authors report diffusion upsamplers cannot afford as many sampling steps as the base model at high resolution |
| Diffusion-level coverage at near-GAN speed | Denoising diffusion GAN or adversarial distillation | Two-step hybrids and 1 to 4 step distilled samplers are published |
| Very little training data | Neither family by default; regularisation decides | Conventional GANs need plenty of data, transformer diffusion failed to converge in a few-shot study, and training strategy mattered more than generator complexity |
| A deepfake detector | Training data from both families | Detectors trained on diffusion images generalise to GANs, but not the reverse |
In our judgement, a hybrid is worth its extra complexity when you need diffusion’s coverage and also have a hard latency budget, as in an interactive face-editing tool. When latency is not the constraint, we would choose plain latent diffusion with a reduced-step sampler. A pure GAN remains the sensible choice for a narrow, well-covered domain where single-pass speed and latent editing matter more than covering every mode.
The unsettled part: one-step quality at deepfake resolution
Adversarial Diffusion Distillation reports beating few-step GANs in a single step. GigaGAN reports being faster than progressively distilled Stable Diffusion, with better FID. Both are results reported by each method’s own authors. Until a neutral comparison exists, the defensible approach is to fix your target resolution and latency first. Then measure recall alongside FID for each candidate at the step count you can actually afford, because quality figures quoted at 50 or 250 steps tell you little about a model that has to run in two.
