A diffusion model in AI is a generative model trained to turn random noise into data one denoising step at a time. In training, noise is slowly added to real examples and a network learns to reverse that process. At generation time the model starts from noise and runs the learned reversal until a data sample, such as an image, comes out. Diffusion is a crucial component of mainstream generators such as Stable Diffusion, DALL-E and Imagen. A text-to-image model takes a natural-language prompt and produces a matching image, and such models are generally latent diffusion models. The goal of a diffusion model, stated precisely, is to learn a reverse process that turns an easy-to-sample Gaussian prior back into a distribution that resembles the training data.
Why they are called diffusion models: the forward noising process
The name comes from physics. Sohl-Dickstein and colleagues (2015) took their core idea from non-equilibrium statistical physics: systematically and slowly destroy the structure in a data distribution through an iterative forward diffusion process, then learn a reverse diffusion process that restores it.
The forward process is a Markov chain of small steps, each adding a little random noise. After enough steps the data distribution has been diffused into a standard Gaussian. In the DDPM formulation, a noisy sample at step t is drawn from a Gaussian centred on a scaled copy of the original data, and as the scaling term approaches zero at the final step the distribution converges to N(0, I). Ho et al. (2020) set the per-step noise variances to a linearly increasing sequence from 10⁻⁴ to 0.02, and Nichol and Dhariwal (2021) later proposed a cosine-based schedule to improve likelihood.
The forward process has no learnable parameters, which means nothing in it needs training. Building it amounts to successively injecting noise into the data; all of the difficulty sits in the reverse direction.
How a diffusion model learns to reverse the noise
Training a denoising diffusion probabilistic model (DDPM) comes down to a noise-prediction regression. Take a clean example from the dataset, pick a random step k, add noise of the variance that step calls for, and ask the network to predict the noise that was added, scoring it with mean squared error. Chi et al., restating Ho et al. (2020), note that minimising this loss also minimises a variational bound on the divergence between the data distribution and the model’s samples. Predicting the noise term rather than the clean image is a reparameterisation; the network still ends up approximating each reverse step.
The original DDPM paper by Ho, Jain and Abbeel (2020) obtained its best results by training on a weighted variational bound built on a connection between diffusion models and denoising score matching with Langevin dynamics. Song and Ermon (2019) had perturbed data with Gaussian noise at many levels, trained a single network conditioned on the noise level to estimate the gradient of the data density at each level, and generated samples by starting at the highest noise level and gradually annealing down.
How does a diffusion model turn random noise into a finished image?
Generation starts from random Gaussian noise, and at each step the trained network predicts the noise it contains. A scheduler then uses that prediction to compute a slightly less noisy version, and the loop repeats until the noise is gone. Each update subtracts the predicted noise, rescales, and in DDPM-style samplers adds a small amount of fresh Gaussian noise back in. In Stable Diffusion the denoising is repeated about 50 times before the result is decoded into an image.
Worked example: one 512×512 Stable Diffusion image, step by step
The sequence below follows the Hugging Face Diffusers walkthrough of Stable Diffusion.
- Encode the prompt into text embeddings; these will condition the denoising network.
- Encode an empty prompt (the padding token) to get unconditional embeddings of the same shape, needed for classifier-free guidance.
- Draw the starting noise in latent space, with shape 1×4×64×64, which is far smaller than the 512×512 image it will become.
- Set the scheduler’s number of inference steps, which fixes the timesteps and noise levels the loop will visit.
- At each timestep, run the U-Net on the conditional and unconditional inputs, combine the two noise predictions with the guidance scale, and let the scheduler step the latents from x_t to x_{t−1}.
- Repeat for roughly 50 steps.
- Rescale the final latents and decode them into pixels with the variational autoencoder’s decoder.
In this pipeline the diffusion model works on a compressed latent representation, which is the subject of the latent diffusion section below.
How does a text prompt guide what a diffusion model generates?
A prompt steers generation in two ways: its embedding conditions the denoising network, and guidance combines conditional and unconditional predictions. Text-to-image systems typically use a pretrained language or vision-language model to turn the prompt into an embedding, and a diffusion model that generates images conditioned on it.
The embedding gets into the network through cross-attention. Rombach et al. (2022) introduced cross-attention layers into the denoising architecture so that it could take general conditioning inputs such as text or bounding boxes. In the Stable Diffusion U-Net those layers sit in both the encoder and decoder halves, usually between ResNet blocks. Stability AI describes Stable Diffusion 3.5 Large as a Multimodal Diffusion Transformer that uses three fixed, pretrained text encoders. The shift from U-Nets to transformers is covered in our piece on the latest advancements in AI image generation.
Guidance strengthens the steer, trading diversity for fidelity. Dhariwal and Nichol (2021) did this with classifier guidance: train a classifier on noisy images and use its gradients to push sampling towards a target class, trading diversity for fidelity. Ho and Salimans (2022) removed the separate classifier. In classifier-free guidance a conditional and an unconditional model are trained jointly and their score estimates are combined, which gives a similar quality-diversity trade-off. In practice both are one network: the conditioning is randomly dropped during training, so the same model learns both modes.
At sampling time the guided prediction is the unconditional prediction plus the guidance scale times the difference between the conditional and unconditional predictions. A guidance scale of 1 means no classifier-free guidance at all, and the Diffusers walkthrough uses 7.5. With classifier guidance, Dhariwal and Nichol found that raising the scale buys precision at the cost of recall, with the best FID at an intermediate setting. Guidance also has a hardware cost: it needs a conditional and an unconditional forward pass at every step, usually batched together into one call. With classifier-free guidance on, the denoiser processes two inputs per step instead of one.
Text is not the only signal a diffusion model can be conditioned on. ControlNet adds trainable copies of the encoder layers so that generation can also follow edge maps, pose skeletons, depth maps or user scribbles.
What is latent diffusion, and how does Stable Diffusion use a compressed latent space?
Latent diffusion runs the noising and denoising on a compressed representation of the image produced by an autoencoder, rather than on raw pixels. Rombach et al. (2022) gave the reason: pixel-space diffusion models typically consume hundreds of GPU days to train, and inference is expensive because of the sequential evaluations, so they moved diffusion into the latent space of a powerful pretrained autoencoder. The intuition is that most of an image’s bits go to perceptual detail, while its semantic composition survives aggressive compression. An autoencoder strips out the pixel-level redundancy and the diffusion model generates the concepts.
The compression in Stable Diffusion is concrete. Its autoencoder has a reduction factor of 8, so a 3×512×512 image becomes a 4×64×64 latent, a spatial compression of 8 × 8 = 64. This is why 512×512 images can be generated quickly even on 16 GB Colab GPUs. At launch, Stability AI stated that Stable Diffusion ran in under 10 GB of VRAM on consumer GPUs and produced 512×512 images in a few seconds. Later versions kept the approach: Stable Diffusion v2 generates 768×768 images from a 4×96×96 latent.
Rombach et al. report that latent diffusion models reached state-of-the-art inpainting and competitive results in unconditional generation, semantic scene synthesis and super-resolution, while needing far less compute than pixel-space models. Diffusion transformers build on the same latent: DiT splits the noisy latent into patches and processes the resulting sequence of tokens with transformer blocks. For the practical side of running the model, see our guide to AI art generation with Stable Diffusion.
How do diffusion models differ from GANs and VAEs?
Diffusion models generate through many sequential denoising passes and are trained with a stationary, likelihood-based objective. GANs and VAEs are single-step generators, and each has a training weakness of its own. Diffusion trades more compute at sampling time for better samples and distribution coverage. Conditional GANs were widely used for text-to-image generation early on, and since 2020 diffusion models have become the popular option.
| Diffusion model | GAN | VAE | |
|---|---|---|---|
| How it generates | Many sequential denoising passes from noise | Single pass through a generator | Single pass through a decoder |
| Training | Stationary, likelihood-based objective; scales easily | Adversarial; can collapse without careful hyperparameters and regularisers | Relies on a surrogate loss |
| Coverage of the data distribution | Higher recall than GANs at similar perceptual quality | Captures less diversity than likelihood-based models | - |
| Sampling cost | Considerably higher than GANs or VAEs | Low | Low |
Dhariwal and Nichol (2021) showed diffusion models beating state-of-the-art generative models on sample quality. Their diffusion models obtained the best FID on each task in their comparison, and with classifier guidance they substantially outperformed the best GANs on higher-resolution ImageNet while keeping higher recall. Nichol and Dhariwal had already used precision and recall to compare how well DDPMs and GANs cover the target distribution. Weng (2021) lists the limitations of each: unstable adversarial training and lower diversity for GANs, a surrogate loss for VAEs, and the specialised reversible architectures that flow models require.
The bill comes at sampling time. Dhariwal and Nichol note that their models are slower than GANs because they need multiple denoising steps, and that each forward pass also takes 5 to 20 times longer because the networks are larger than the competing GAN generators. One data point Weng quotes from Song et al. (2020): sampling 50,000 32×32 images took around 20 hours from a DDPM, but less than a minute from a GAN on an Nvidia 2080 Ti.
Why is generating with a diffusion model slower and more compute-heavy?
Diffusion generation is a sequence of full network evaluations, not one pass. Every timestep requires a complete forward pass through a large network, often a U-Net variant. The total cost therefore scales linearly with the number of steps, which ranges from tens to thousands depending on the model and sampler. Early diffusion models needed 1,000 or more steps, and newer samplers bring that down to around 20 to 50, which is still substantial next to a single-pass model.
The cost of each step is high too. State-of-the-art diffusion models often contain hundreds of millions to billions of parameters, and loading them requires significant GPU VRAM. Self-attention layers scale quadratically with the number of pixels or patches, so going from 512×512 to 1024×1024 increases FLOPs per step and activation memory, often super-linearly. So the cost of one image grows with the step count, the cost of each step, and the extra pass classifier-free guidance adds.
The fixes so far attack the step count, or the fact that steps run one after another:
- Learned variances. Nichol and Dhariwal (2021) found that learning the variances of the reverse process allows sampling with an order of magnitude fewer forward passes, with a negligible difference in sample quality.
- DDIM. Song, Meng and Ermon (2020) built non-Markovian processes with the same training procedure as DDPMs, producing high-quality samples 10 to 50 times faster in wall-clock time. ODE-based samplers like DDIM tend to sample faster, while SDE-based samplers like DDPM often give higher quality with enough runtime. Cutting the step count of the stochastic DDPM sampler directly hurts sample quality significantly, which is why low-step methods moved to ODE formulations.
- Parallel sampling. ParaDiGMS computes multiple denoising steps in parallel through Picard iterations and gives around 2 to 4 times speedup across image and robotics models without measurable quality loss. It works alongside DDIM and DPMSolver rather than replacing them. On Stable Diffusion its authors saw over 3 times speedup using a batch window of 80 across 8 A100s.
Parallel sampling trades extra compute for faster sampling. Whether that trade is worth making depends on whether you are limited by latency per image or by throughput per GPU, and that should be settled before choosing a sampler.
What diffusion models are used for besides image generation
Denoising diffusion probabilistic models have been applied to image and video generation, protein and material synthesis, weather forecasting and neural surrogates of partial differential equations. Other reported uses include text generation, speech synthesis, time-series imputation, reinforcement learning and molecule modelling. Four applications show how far the same loop travels:
- Video. Imagen Video generates high-definition video from a text prompt using a cascade of video diffusion models: a base model followed by interleaved spatial and temporal super-resolution models. Text-to-video platforms such as Runway, Make-A-Video and Phenaki followed the text-to-image models.
- Weather. GenCast (Price et al., Nature 2024) produces a future atmospheric state by iteratively refining a candidate that starts as pure noise, conditioned on the two previous states. Its noise is sampled on the sphere instead of independently per grid point, to respect the geometry of global weather.
- Protein design. RFdiffusion (Watson et al., Nature 2023) noises structures from the Protein Data Bank for up to 200 steps and fine-tunes a structure prediction network as the denoiser. Because each trajectory starts from random noise, each one yields a different design.
- Robot control. Diffusion Policy (Chi et al.) generates robot actions and can represent multimodal action distributions, a known difficulty in policy learning. It also predicts a sequence of future actions at once rather than one step at a time.
The robotics case shows the cost section’s point in a different setting. Chi et al. state that Diffusion Policy has higher computational cost and inference latency than simpler methods, and that predicting action sequences only partly offsets this for tasks needing high-rate control. Wherever the loop goes, the step count goes with it, so the questions are the same in every domain: how many steps the application can afford, how expensive each step is, and whether the hardware can run them fast enough. For how these architectures match real products, our overview of generative AI applications goes further. TechnoLynx builds and tunes such models into products through its Generative & Agentic AI R&D practice.
