Diffusion models: how image generators turn noise into pictures
Diffusion models are generative neural networks trained to reverse a gradual noising process. They add noise to training images step by step, then learn to denoise starting from pure noise to synthesize new images. This approach powers many modern text-to-image generators and related computer vision tasks.
What diffusion models do
Diffusion models are a class of generative models that learn to create data by reversing a gradual noise addition process. According to IBM's overview, they are used mainly for image generation and related tasks such as inpainting and super-resolution. The intuition comes from physics: as ink disperses in water, pixels spread into random noise over time.
The models train on samples from a data distribution, progressively adding random noise until the image becomes pure static. Then a neural network is trained to run that process in reverse, turning random noise back into a plausible image. This learned denoising capability allows the model to generate new samples that resemble the training data.
Diffusion models have become prominent in generative AI because they produce high-quality results and offer a stable training process compared to other generative architectures like GANs or VAEs, though the IBM page notes that they were independently discovered through two mathematical approaches.
The forward and reverse processes
The forward diffusion process was introduced in the nonequilibrium thermodynamics paper, which described systematically destroying structure in a data distribution through iterative noise addition. Starting from a clean image, each step adds a small amount of Gaussian noise according to a variance schedule, eventually yielding near pure noise.
The reverse process is learned as a denoising diffusion probabilistic model, as detailed in the DDPM paper. This paper demonstrated high-quality image synthesis by training a neural network to predict the noise that was added, enabling the network to gradually remove noise from a random sample.
A practical implementation is explained in the annotated diffusion model, which shows that the process typically runs for a fixed number of time steps, often 1000. The forward process uses a defined schedule for noise levels, and the network is optimized to estimate the noise at each step.
Score-based perspective and unified framework
Diffusion models have a close connection to score-based generative models, which estimate the gradient of the log probability density of data. The score-based SDE framework shows that a stochastic differential equation can smoothly transform a data distribution into a known prior by injecting noise, and a corresponding reverse-time SDE can transform it back.
This framework encapsulates earlier diffusion and score matching approaches, enabling new sampling procedures and exact likelihood computation through an equivalent neural ODE. The paper also demonstrates high-fidelity generation of large images and introduces methods for solving inverse problems.
Latent diffusion for efficiency
To reduce computational demands, the latent diffusion model paper proposes applying diffusion in the latent space of a pretrained autoencoder rather than in pixel space. This allows high-resolution synthesis while significantly lowering training and inference costs.
The same paper introduces cross-attention layers into the model architecture, enabling conditioning on inputs such as text or bounding boxes. This makes latent diffusion models flexible generators for text-to-image and related tasks, as the model can attend to the conditioning signal during denoising.
Practical implementation with libraries
For developers and researchers, the Hugging Face Diffusers documentation provides pretrained diffusion models and modular building blocks for image, video, and audio workflows. It offers two main paths: a simple DiffusionPipeline for few-line inference and a more modular approach for custom pipelines.
The documentation also covers optimizations such as offloading, quantization, and torch.compile, which help run large models on memory-constrained devices. This makes diffusion models accessible for a wide range of applications and hardware.
| Step | Description |
|---|---|
| Forward diffusion | Gradually adds Gaussian noise to a clean image according to a variance schedule. |
| Noise schedule | Defines the amount of noise at each time step, often linear or cosine. |
| Reverse denoising | A neural network learns to predict and remove the noise from a noisy sample. |
| Training objective | Optimizes a loss that measures how well the network estimates the added noise. |
| Generation | Starts from pure random noise and applies the reverse process to create a new image. |
Frequently asked questions
How does a diffusion model generate an image?
It starts with a sample of pure random noise and repeatedly applies a learned denoising step, gradually revealing a structure that resembles the training data.
What is the forward diffusion process?
The forward process adds increasing amounts of Gaussian noise to a clean image over many time steps, eventually turning it into isotropic Gaussian noise.
Why are latent diffusion models more efficient?
They perform the diffusion process in the latent space of a pretrained autoencoder, which has lower dimensionality than pixel space, reducing computational requirements while preserving visual fidelity.
What is the score function in score-based models?
The score is the gradient of the log probability density of data. Score-based models learn to estimate this gradient, which enables reversing the noise injection process.
Can diffusion models be used for tasks other than images?
Yes, diffusion models are also applied to audio synthesis, drug design and molecule generation, as noted in IBM's overview of the topic.