Diffusion models generate images, audio or video by learning to reverse a gradual noising process. They are built from two processes: a fixed forward process that adds Gaussian noise to real data, a little at a time, until nothing but noise is left, and a learned reverse process in which a neural network removes that noise one step at a time. To generate something new, the model draws a sample of pure Gaussian noise and runs the reverse chain until structured data comes out.

Sohl-Dickstein and colleagues described it in 2015 as slowly destroying structure in a data distribution through an iterative forward diffusion process, then learning a reverse process that restores it. The 2020 DDPM paper by Ho, Jain and Abbeel made it practical. It used fixed variances and a simple L2 loss on the added noise, showed results on natural images that rivalled GANs, and became the baseline later work built on.

Why the forward process adds noise in small steps

The forward process is a Markov chain with T transitions. At step t it takes the previous sample and produces a slightly noisier one, using the Gaussian kernel q(x_t

x_{t-1}) = N(x_t; √(1 − β_t) x_{t-1}, β_t I), where β_t is a value between 0 and 1 set in advance that controls how much noise goes in at that step. As t increases, the noise adds up until the distribution of x_T is close to an isotropic Gaussian. If the chain structure is new to you, our explainer on Markov chains in generative AI covers the memoryless property this relies on.

The scaling factor √(1 − β_t) has a purpose. Shrinking the previous sample while adding noise keeps the overall variance from exploding or vanishing over many iterations. This is the variance-preserving form of diffusion, and it matters because changing the input’s statistics as noise accumulates can affect learning.

The step size matters for a deeper reason too. When each β_t is small enough, the true reverse step q(x_{t-1}

x_t) is also Gaussian. That result goes back to Feller (1949): in the limit of infinitesimal steps, the reverse of a diffusion has the same distributional form as the forward process. So each reverse step is parameterised as a Gaussian. It only has to estimate the mean (and possibly the variance) of a Gaussian at each step. Many small, simple steps replace one hard jump from noise to data.

There is also a shortcut that makes training cheap. Define ᾱ_t as the product of (1 − β_s) for s from 1 to t. The noisy sample at any step can then be written directly from the clean one: q(x_t

x_0) = N(x_t; √ᾱ_t x_0, (1 − ᾱ_t) I), with ᾱ_t approaching zero by the final step so that x_T is effectively pure noise. Training never has to walk the chain: it can jump straight to any timestep. The forward process involves no learning at all. Given a clean image, x_1 to x_T can be prepared with no trained parameters.

What does the network actually learn to predict when it removes noise?

In the standard DDPM setup, the network is trained to predict the noise that was added. Given a noisy sample x_t and the timestep t, it outputs an estimate ε_θ(x_t, t) of the Gaussian noise ε used to make x_t, and the loss is the squared error ‖ε − ε_θ(x_t, t)‖² with t drawn uniformly from the diffusion steps.

The true reverse step on its own, q(x_{t-1}

x_t), cannot be estimated easily, because it would need the entire dataset. Conditioned on the clean sample x_0, though, the posterior q(x_{t-1}

x_t, x_0) is a Gaussian with a closed form, a consequence of Gaussian conjugacy. That posterior needs no network at all. It is fully determined once x_t and x_0 are known. The learned reverse step is a Gaussian whose mean a network predicts, matched to that known posterior. Ho et al. (2020) parameterised the mean through the noise estimate, which separates reconstructing the data from estimating the noise and trains more stably than predicting the mean directly.

The full objective is a variational bound made of one KL divergence term per timestep. Ho et al. (2020) found empirically that dropping the weighting terms and keeping the plain noise-regression loss gave better results.

One training step, in order

1. Take a clean sample x_0 from the training set.
2. Draw a timestep t uniformly from 1 to T.
3. Draw Gaussian noise ε ~ N(0, I).

4. Form x_t in one shot from x_0 and ε using the closed-form q(x_t

   x_0).

5. Feed x_t and t to the network and compute ‖ε − ε_θ(x_t, t)‖².

Sampling then runs the other way. It starts from x_T ~ N(0, I) and repeatedly samples x_{t-1} from the learned Gaussian until x_0, a procedure known as ancestral sampling.

Different timesteps carry different noise levels, so the network takes t alongside x_t to know how much to undo. One network covers every noise level, because training a separate denoiser per level would not be feasible computationally. In practice that network has been a time-conditioned U-Net, and more recently a Transformer operating on patches of the latent, the Diffusion Transformer design.

Noise is not the only possible target. The network can predict the clean image x_0, the noise ε or a velocity v, and each can be reparameterised into the same posterior mean. These targets are consistent with each other once the loss weights are chosen coherently. Even so, switching to v prediction is one of the fixes Lin et al. (2023) propose for a schedule flaw covered below. Noise prediction also connects to a second formulation. The predicted noise, scaled by −1/√(1 − ᾱ_t), estimates the score function, the gradient of the log density of the noisy data.

How the noise schedule shapes the forward and reverse process

The noise schedule is the sequence β_1 to β_T. It sets how quickly the forward process destroys the data. Ho et al. (2020) used a linear schedule running from β_1 = 10⁻⁴ to β_T = 0.02, small values relative to image pixels normalised to [−1, 1].

Nichol and Dhariwal (2021) found that the linear schedule worked well for high-resolution images but was sub-optimal at 64×64 and 32×32. At those sizes the end of the forward process was too noisy to contribute much to sample quality. Their cosine schedule drops ᾱ_t roughly linearly through the middle of the process and changes it very little near t = 0 and t = T, which avoids abrupt jumps in noise level. It also improved log-likelihoods.

Chen (2023) went further and found that the schedule is crucial to performance and that the best schedule depends on the task. As image size increases, the optimal schedule shifts towards a noisier one, which Chen attributes to increased pixel redundancy. Chen also found that scaling the input data by a constant while keeping the schedule function fixed works well across image sizes. Both findings tie the best schedule to image size. For that reason we would treat the schedule as a hyperparameter to revisit whenever the resolution or data domain changes, not a default to inherit from a reference implementation.

The schedule does not have to be hand-set. Variational diffusion models train the schedule jointly with the network to maximise the variational bound, and in continuous time that bound is invariant to the schedule. The schedule can then be adapted to reduce the variance of the loss estimate.

A known failure: non-zero terminal SNR

Lin et al. (2023) found that common noise schedules do not drive the signal-to-noise ratio at the last timestep to zero, and that some samplers do not start from the last timestep. The model is given pure Gaussian noise at inference, so training and inference disagree. Their fixes are to rescale the schedule to zero terminal SNR, train with v prediction, always start sampling from the last timestep, and rescale classifier-free guidance to prevent over-exposure.

The main types of diffusion models and how they relate

DDPM, score-based models and score SDEs share the same principle of perturbing data with noise and then removing it.

Score-based generative models (SGMs) learn the score, ∇ log q(x), and generate samples with Langevin dynamics. Song and Ermon (2019) perturbed data at several noise levels and trained one noise-conditioned score network across all of them, a schedule of rising noise that mirrors the forward diffusion process. Ho et al. (2020) showed an equivalence between DDPM and score-based models, and the survey by Yang et al. shows that the two training objectives match under a simple rescaling of the network output.

Score SDEs take both to continuous time. Song et al. (2021) describe a stochastic differential equation that turns data into a known prior by slowly injecting noise, and a reverse-time SDE that depends only on the score of the perturbed data. The framework includes earlier score-based and diffusion probabilistic approaches as special cases. It also yields a probability-flow ODE, a deterministic process with the same marginal distribution at every time as the stochastic one. Of the SDE choices, variance-preserving (VP) and variance-exploding (VE) are the most widely used; Stable Diffusion uses VP and consistency models rely on VE.

DDIM (Song, Meng and Ermon, 2020) keeps the DDPM training procedure but samples with a non-Markovian process that reaches the same objective and can be sampled much faster. Its per-step variance is free: setting it to DDPM’s posterior variance recovers DDPM, and setting it to zero gives a deterministic sampler. That deterministic sampler is a discretisation of the probability-flow ODE, and the authors report high-quality samples 10× to 50× faster in wall-clock time than DDPM.

Latent diffusion (Rombach et al., 2022) first trains an autoencoder to produce a low-dimensional latent space, then trains the diffusion model to generate latent codes. The forward marginals, posterior and DDPM/DDIM variance choices all carry over unchanged once x is replaced by the latent z, which is why the speedup comes without new mathematics. Cross-attention layers let it condition on text or bounding boxes, and running in latent space lowers training cost and speeds up inference. Stable Diffusion is the best-known example, and our piece on how AI image generators work follows that pipeline from prompt to pixels.

Flow matching (Lipman et al., 2022) trains continuous normalising flows by regressing vector fields along fixed conditional probability paths, with no simulation. Its family of Gaussian paths includes diffusion paths as special cases, and the authors report that flow matching with diffusion paths trains diffusion models more robustly and stably. Rectified flow connects data and noise in a straight line, and it recovers DDIM up to a reparameterisation of time. The two views are closer than the names suggest: training a noise predictor is equivalent, up to an explicit linear transform, to learning the rectified velocity.

Variant	What it changes	Why it exists
DDPM	Discrete Markov chains, fixed variances, L2 noise loss	Practical baseline for image quality
iDDPM	Learned reverse variances, hybrid objective, cosine schedule	Better likelihood and far fewer sampling passes
SGM / NCSN	Learns the score at several noise levels	Score matching view of the same objective
Score SDE	Continuous-time forward and reverse SDEs	Unifies DDPM and SGM, adds the probability-flow ODE
DDIM	Non-Markovian, optionally deterministic sampler	Faster sampling with the same training
Latent diffusion	Diffuses in an autoencoder’s latent space	Cuts compute at high resolution
Flow matching / rectified flow	Regresses a velocity field along chosen paths	Simulation-free training and ODE sampling
Consistency models	Map noise to data in one step	One-step sampling

Why diffusion sampling is slow, and what reduces the step count

Sampling from a diffusion model is slow because the reverse steps must run sequentially. Every DDPM denoising step has to wait for the previous one, which is a significant disadvantage next to generative models whose processes can be parallelised. One data point comes from Song et al. (2020): generating 50,000 images of 32×32 from a DDPM took around 20 hours, against less than a minute for a GAN, on an Nvidia 2080 Ti. The imaging and vision tutorial, citing Song et al., puts 50,000 images at 256×256 at more than 1,000 hours on a standard GPU. Nichol and Dhariwal’s models were trained with 4,000 diffusion steps, so a single sample took several minutes on a modern GPU.

One main source of the slowness is the Gaussian assumption. A Gaussian is a good model for the reverse step only when the step is small, so the process needs many of them. Acceleration methods fall into two groups: those that need no extra learning and those that train something more after the diffusion model is done.

• Better samplers, no retraining. DDIM is one of the earliest of these. Higher-order ODE solvers that exploit the semi-linear structure of the probability-flow ODE (DDIM is their first-order case) produce high-quality samples in 10 to 20 iterations, against the hundreds typical without acceleration.
• Learned variances. Learning the reverse variances lets a model sample with an order of magnitude fewer forward passes at a negligible cost in quality.
• Redesigned training and sampling. Karras et al. (2022) separated the design choices and reached an FID of 1.79 on class-conditional CIFAR-10 at 35 network evaluations per image.
• Distillation and one-step models. Consistency models sample in one step, and consistency distillation trains them from a pretrained score model.
• Non-Gaussian reverse steps. Denoising diffusion GANs model each step with a multimodal distribution learned adversarially, so they can take large steps.

One property of DDIM decides where we would start on a slow diffusion pipeline. The training objective is the same for any choice of sampler variance, so a model trained with the DDPM process can be sampled with DDIM without retraining. Changing the sampler and step count leaves the weights untouched. We would therefore try those first, then compare output quality against latency on real prompts before considering distillation, which needs another training run.

How guidance steers the reverse process towards a prompt

Guidance changes the reverse transitions so that sampling moves towards a goal such as a class label or a text prompt. The forward process and the training framework stay as they are. Classifier guidance, introduced by Dhariwal and Nichol (2021), adds the gradient of a separately trained classifier to the reverse mean. The authors describe it as a simple, compute-efficient way to trade diversity for fidelity.

Classifier-free guidance drops the classifier. One network is trained on (image, caption) pairs, and the caption is randomly suppressed during training, so the same model learns both conditional and unconditional noise predictions. At sampling time the two predictions are blended to push the result towards the condition. Lin et al. (2023) include rescaling classifier-free guidance to prevent over-exposure among their fixes.

What diffusion models are used for beyond still images

The same forward and reverse machinery has been extended to video, audio, discrete data and hyperspectral imaging. Ho et al. (2022) built a video diffusion model as a natural extension of the standard image architecture. Training jointly on images and video reduced the variance of minibatch gradients and sped up optimisation, and a conditional sampling technique extended videos in space and time. The same paper reported early results on large-scale text-conditioned video generation.

In audio, DiffWave (Kong et al., 2020) is a non-autoregressive diffusion model. It turns white noise into a waveform through a Markov chain with a fixed number of steps and is trained on a variant of the variational bound. WaveGrad makes the step-count trade-off explicit, giving up inference speed for sample quality as refinement steps are added.

Text and other discrete data are a harder fit. Gaussian noise does not suit discrete values, and the score is only defined on continuous domains. Discrete variants therefore replace Gaussian noise with absorbing-state or discretised Gaussian kernels (D3PM), or with a random walk or random masking (VQ-Diffusion). In hyperspectral imaging, the stepwise denoising preserves continuous spectral signatures that single-step generators such as GANs tend to distort. Our article on diffusion models beyond images goes further into audio, protein and tabular uses.

How diffusion models differ from GANs, VAEs, flows and LLMs

Diffusion models give up sampling speed in exchange for stable training and good coverage of the data distribution. A GAN trains a generator against a discriminator. It generates quickly, but training is unstable and can suffer mode collapse. Diffusion models are described as the only framework that rivals GANs in most applications without adversarial training. Splitting generation into many conditional steps keeps each step simple, which avoids the difficulty GANs have in producing a complex distribution in a single shot. Measured by recall, Nichol and Dhariwal (2021) found that diffusion models covered the modes of the distribution better than a comparable BigGAN-deep, even where the GAN scored better on FID.

The link to VAEs runs deeper than a contrast. DDPM can be read as a hierarchical Markovian VAE with a fixed encoder: the forward process is the encoder and is not learned, and one shared network acts as the decoder. The latent variables are all the same size as the data. Because the final noise distribution equals the prior by construction, diffusion models avoid the “hole” problem in VAEs, where the encoded data and the prior do not line up. VAEs, by comparison, tend to produce blurry outputs. Normalising flows need specialised reversible architectures to keep their transform invertible.

The trade-offs also cut against diffusion. Its latent space is less useful as a representation than a VAE’s or a GAN’s, so semantic editing is harder, and a latent with the same dimensionality as the data hurts sampling efficiency. Even with the newer acceleration methods, sampling remains slower than a GAN.

Large language models are mostly autoregressive: they generate one token at a time, each conditioned on the tokens before it. DiffWave, for example, is explicitly non-autoregressive. Ho et al. (2020) note that DDPM sampling can be read as progressive lossy decompression that generalises autoregressive decoding. Another obstacle to diffusion for text is the data type, since Gaussian noise is not a natural fit for discrete data.

Model family	How it generates	Training	Latent	Sampling cost
GAN	Generator maps noise to data in one pass	Adversarial, can be unstable, mode collapse	Better representation than diffusion’s	Fast
VAE	Decoder maps a sampled latent to data	ELBO, learned encoder	Learned by an encoder	Outputs can be blurry
Normalising flow	Invertible transform of a simple distribution	Specialised reversible architecture	Same size, reversible
Autoregressive	One element at a time, in order	Next-element prediction	None	Sequential per element
Diffusion	Many denoising steps from Gaussian noise	Stable regression on noise	Same size as data, fixed encoder	Many sequential steps

For a team choosing a generator, we would weigh tolerable latency per sample and coverage of the data’s full range above architecture preference. We would favour diffusion where diversity and fidelity matter and offline generation is acceptable. A real-time path with a narrow output distribution makes the GAN’s single pass hard to give up, unless a distilled or few-step diffusion sampler closes the gap. Our GAN vs diffusion architecture comparison works through that choice case by case.

FAQ

Who invented diffusion models, and when?

Sohl-Dickstein et al. introduced diffusion probabilistic models in 2015, drawing on non-equilibrium thermodynamics. The 2015 paper demonstrated the idea on simple datasets, and the models only became competitive on images after Ho et al. (2020) and Song et al. built practical frameworks.

Is diffusion in AI based on the random movement of particles?

Only loosely. Diffusion models are loosely modelled on non-equilibrium thermodynamics, where diffusion means particles flowing from high to low concentration, and the statistical version transforms a complex data distribution. The forward process adds Gaussian noise to the data.

Sources

1. Improved Denoising Diffusion Probabilistic Models (Nichol & Dhariwal, 2021), ar5iv.labs.arxiv.org
2. Diffusion Models: A Mathematical Introduction, arxiv.org
3. Video Diffusion Models (Ho et al., 2022), arxiv.org
4. Diffusion Models: A Comprehensive Survey of Methods and Applications, arxiv.org
5. Deep Unsupervised Learning using Nonequilibrium Thermodynamics (Sohl-Dickstein et al., 2015), arxiv.org
6. Diffusion Models for Hyperspectral Image Analysis: A Comprehensive Review, arxiv.org
7. Diffusion Models in 3D Vision: A Survey, arxiv.org
8. Tutorial on Diffusion Models for Imaging and Vision, arxiv.org
9. Demystifying Variational Diffusion Models, arxiv.org
10. What are Diffusion Models? (Lilian Weng), lilianweng.github.io
11. Lecture Notes in Probabilistic Diffusion Models, arxiv.org
12. Common Diffusion Noise Schedules and Sample Steps are Flawed (Lin et al., 2023), arxiv.org
13. On the Importance of Noise Scheduling for Diffusion Models (Chen, 2023), arxiv.org
14. Score-Based Generative Modeling through Stochastic Differential Equations (Song et al., 2021), arxiv.org
15. Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial, arxiv.org
16. Denoising Diffusion Implicit Models (Song, Meng, Ermon, 2020), arxiv.org
17. High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al., 2022), arxiv.org
18. Flow Matching for Generative Modeling (Lipman et al., 2022), arxiv.org
19. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (Esser et al., 2024), arxiv.org
20. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (Xiao et al., 2022), ar5iv.labs.arxiv.org
21. Elucidating the Design Space of Diffusion-Based Generative Models (Karras et al., 2022), arxiv.org
22. Diffusion Models Beat GANs on Image Synthesis (Dhariwal & Nichol, 2021), arxiv.org
23. DiffWave: A Versatile Diffusion Model for Audio Synthesis (Kong et al., 2020), arxiv.org
24. Denoising Diffusion Probabilistic Models (Ho, Jain, Abbeel, 2020), arxiv.org