How Diffusion Models Work: Stable Diffusion, DALL-E, and the Science of AI Art

Diffusion models are the technique behind many of today's AI image generators, including Stable Diffusion and DALL-E 2. The idea is to take a clean image, bury it in random noise through many small steps, and train a neural network to undo each step. Once the network can do that, you can start from pure static and let it dig an image out, steering it with a text prompt along the way. This article explains how diffusion models work, from the noise schedule to the U-Net to the compression trick that let Stable Diffusion run on a home GPU.

The big picture: learning to un-destroy images

Put a drop of ink in a glass of water. At first the ink is a tight, structured blob. Over minutes it spreads until the water is uniformly tinted, and the original shape is gone. Physicists call this diffusion, and it is a one-way street: ink never gathers itself back into a drop.

Machine learning researchers asked an odd question. If you recorded the process in very small steps, could you train a model to run it backward? In 2015, Jascha Sohl-Dickstein and colleagues proposed exactly that in a paper titled "Deep Unsupervised Learning using Nonequilibrium Thermodynamics." Their method gradually corrupts a data distribution with a forward diffusion process, then learns a reverse process that restores the structure. The result is a generative model: start from noise, run the reverse process, and you get something that looks like the training data.

The key insight is that each backward step is easy. Removing a faint dusting of noise from an almost-clean image is a much smaller task than painting a picture from a blank canvas. A diffusion model never has to solve the hard problem in one move. It solves a thousand small ones in sequence. For readers who want the thermodynamic background, our explainer on entropy and the arrow of time covers why diffusion only runs one way in nature.

A short timeline of the idea

The technique took about seven years to go from a physics-flavored paper to a consumer product. The table lists the milestones that matter for understanding the rest of this article.

Year	Milestone	Why it mattered
2015	Sohl-Dickstein et al., nonequilibrium thermodynamics paper	First diffusion-based generative model
2019	Song and Ermon, score-based generative modeling	Sampling by following learned gradients across several noise levels
2020	Ho, Jain, and Abbeel, DDPM	Simple training objective; FID of 3.17 on CIFAR-10
2020	Song, Meng, and Ermon, DDIM	Samples 10 to 50 times faster than DDPM
2021	Dhariwal and Nichol, "Diffusion Models Beat GANs"	Better ImageNet sample quality than the best GANs
2022	Latent diffusion, DALL-E 2, Imagen, Stable Diffusion	Text-to-image at scale; open weights
2023	DiT, ControlNet, consistency models	Transformer backbones, spatial control, one-step sampling
2024	Stable Diffusion 3 paper, Sora report	Rectified flow, transformers on latent patches, video

Each row is a published paper or announcement, and the sources are listed at the end.

The forward process: adding noise step by step

The forward process (also called the diffusion process) corrupts a clean image x₀ by adding a small amount of Gaussian noise at each step. Gaussian noise is the static you get when every pixel is nudged by an independent random amount drawn from a bell curve. After enough steps nothing of the original remains.

In the DDPM paper, the number of steps is T = 1,000, and the amount of noise added at each step (written β) rises linearly from 0.0001 at step 1 to 0.02 at step 1,000. The authors chose those constants so that, by the last step, the image is statistically indistinguishable from pure noise.

You do not need to run a thousand steps to corrupt an image. Because the noise is Gaussian, there is a closed-form shortcut that jumps straight to any step t:

x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε

Here ε is a fresh sample of Gaussian noise, and ᾱ_t (alpha-bar) is the running product of (1 − β) over all steps up to t. It measures how much of the original signal survives. The first term is the faded original image, and the second is the noise that has replaced it.

How fast does the picture disappear?

Using the linear schedule from the DDPM paper, I computed ᾱ_t at a few checkpoints. The values are rounded, and they show that the image does not fade evenly.

Step t	ᾱ_t (signal left)	Signal amplitude √ᾱ_t	Noise amplitude √(1 − ᾱ_t)
0	1	1.00	0.00
100	about 0.90	about 0.95	about 0.32
250	about 0.52	about 0.72	about 0.69
500	about 0.08	about 0.28	about 0.96
750	about 0.003	about 0.06	about 1.00
1,000	about 0.00004	about 0.006	1.00

By the halfway point, noise outweighs the picture by more than three to one in amplitude, though a faint outline remains. By step 1,000, the original contributes less than 1 percent of the amplitude. The forward process is fixed arithmetic, with nothing to learn. What it produces is a training set: for every clean image, you can manufacture a noisy version at any of 1,000 noise levels, along with the exact noise you added.

Think of it as photocopying a photocopy. The first copy looks fine, the tenth looks grainy, and by the thousandth you have static. Diffusion training uses the whole chain as study material.

The reverse process: learning to remove noise

The reverse process is the part that is learned. A neural network receives a noisy image x_t and the step number t, and predicts the noise ε that was mixed in. In symbols, the network ε_θ(x_t, t) tries to match ε.

The training loop in the DDPM paper (its Algorithm 1) is short enough to describe in four lines:

1. Pick a clean training image x₀.
2. Pick a random step t between 1 and T.
3. Draw fresh Gaussian noise ε and build the noisy image √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε.
4. Take a gradient step to reduce the squared difference between ε and the network's prediction.

The loss is a mean squared error between actual noise and predicted noise. If you want to know more about how those gradient steps work, see our explainer on how gradient descent finds better answers. The "simplified" loss used here discards some weighting terms from the full variational bound, and the authors found that this gave better sample quality. In their ablation, predicting the noise with the simplified loss reached an FID of 3.17, against 13.22 and 13.51 for the variants trained with the full variational bound and fixed variances.

Why predicting noise works

Predicting the noise is equivalent to predicting which direction in image space leads back toward a cleaner image. The DDPM authors note that their objective resembles denoising score matching over multiple noise scales, which links the method to the score-based approach Yang Song and Stefano Ermon published in 2019. In that view, the network learns a map of "which way is more image-like" at every noise level, and generation follows that map downhill.

Picture a hiker in thick fog trying to find a valley. The hiker cannot see the valley, but can feel the slope underfoot. At each step, the network reports the slope, and the sampler takes a small step in that direction. A thousand small steps get the hiker to the bottom.

Generating an image

Generation runs the process in reverse, following the DDPM paper's Algorithm 2:

1. Start with pure Gaussian noise x_T.
2. For t from T down to 1, ask the network to predict the noise in x_t, subtract a scaled version of it, and (except at the last step) add a small amount of fresh noise.
3. After the final step, x₀ is the generated image.

Every starting noise produces a different image, which is where the variety comes from. The original DDPM needs all 1,000 steps, so each image takes 1,000 passes through the network.

The U-Net: the network that does the denoising

The workhorse behind the original DDPM and the first generation of Stable Diffusion is a U-Net, an architecture first published in 2015 by Olaf Ronneberger, Philipp Fischer, and Thomas Brox for biomedical image segmentation. Its abstract describes "a contracting path to capture context and a symmetric expanding path that enables precise localization." Drawn on paper, those two paths form the letter U.

The left side of the U compresses the image, step by step, into smaller and more abstract feature maps that capture large-scale structure. The right side expands back to full size. Skip connections copy detailed features from each compressing stage straight across to the matching expanding stage, so fine detail is not lost on the way through the bottleneck. A denoiser needs both views: global layout to know what the picture is, and local texture to clean it up.

The DDPM paper's version uses a U-Net backbone similar to the PixelCNN++ design, with group normalization throughout. It feeds the step number t into the network through a sinusoidal embedding, the same trick transformers use to encode position, so the same weights can handle every noise level. It also adds self-attention at the 16 × 16 feature map, which lets distant parts of the image coordinate. A face generated without that coordination might get two different eye colors, and attention is what keeps the left half consistent with the right.

Transformers are replacing the U-Net

The U-Net is not a law of nature. In 2022, William Peebles and Saining Xie published "Scalable Diffusion Models with Transformers," which replaces the U-Net in a latent diffusion model with a transformer that operates on patches of the latent image. Their largest model, DiT-XL/2, reached an FID of 2.27 on class-conditional ImageNet at 256 × 256 resolution, which the abstract describes as state of the art at the time. For background on the attention mechanism these models rely on, see how transformers pay attention to words.

Stable Diffusion 3 and OpenAI's Sora follow this direction. According to the Stable Diffusion 3 paper, the model uses a transformer with separate weights for text and image tokens. OpenAI's February 2024 technical report describes Sora as a transformer that works on spacetime patches of compressed video.

Text conditioning: connecting words to pixels

An unconditional diffusion model generates plausible images, but you cannot ask it for anything in particular. The text-to-image systems add a second input: a numeric representation of the prompt. How that representation is built, and how it steers the denoiser, is where DALL-E, Imagen, and Stable Diffusion differ.

CLIP: a shared space for pictures and captions

CLIP (Contrastive Language-Image Pre-training) is a model from OpenAI, described in a 2021 paper by Alec Radford and colleagues. It was trained on 400 million image-text pairs collected from the internet. The training goal is contrastive: for a batch of images and captions, make the numeric vector (embedding) of each image close to the vector of its own caption and far from the others.

The result is a shared space in which an image of a dog and the phrase "a photo of a dog" land near each other. For a deeper look at how such vectors work, see our guide to embeddings and vector search.

Three ways to feed the prompt into a diffusion model

The systems below all use diffusion, but they inject text differently. The details come from each system's paper or repository.

System	How text enters	Source
DALL-E 2	A "prior" turns the caption into a CLIP image embedding; a diffusion decoder generates the image from that embedding	Ramesh et al., April 2022
Imagen	A text-only language model (T5) encodes the prompt; the authors found scaling the language encoder helped more than scaling the image model	Saharia et al., May 2022
Stable Diffusion v1	A frozen CLIP ViT-L/14 text encoder (123 million parameters); the U-Net attends to its output through cross-attention	CompVis repository; Rombach et al., 2022

Notice that "uses CLIP" means different things here. DALL-E 2 uses CLIP's image embedding space as an intermediate target, while Stable Diffusion v1 uses only the text half of CLIP as an off-the-shelf reader. Midjourney has not published its architecture, so statements about how it works internally are guesses.

Cross-attention

In latent diffusion models, the text conditioning works through cross-attention layers inserted into the U-Net, according to the Rombach et al. abstract. At each spatial position, the network looks at the whole sequence of text embeddings and pulls in whichever words are relevant to what it is drawing there. The word "red" influences the region where the cube is, and "sphere" influences the region where the sphere is. This is the same attention mechanism used in language models, with queries coming from the image and keys and values coming from the text.

Classifier-free guidance

Conditioning alone tends to produce images that only loosely match the prompt. The fix is classifier-free guidance, introduced by Jonathan Ho and Tim Salimans in July 2022. Earlier guidance methods needed a separate image classifier. Ho and Salimans showed that you can jointly train a conditional and an unconditional model and combine their predictions, so that guidance can be done "by a pure generative model without such a classifier."

In practice, one network is trained with the text prompt randomly dropped some fraction of the time. The Stable Diffusion v1-3 and v1-4 checkpoints, for example, used a 10 percent text-conditioning dropout, according to the CompVis repository. At sampling time, the network runs twice per step, once with the prompt and once without, and the two predictions are combined:

ε_guided = ε_unconditional + w · (ε_conditional − ε_unconditional)

The guidance scale w controls how hard the sampler pushes toward the prompt. A higher value gives closer prompt adherence but less diversity. Papers and software differ on whether the scale is written as w or 1 + w, so a "7.5" in one tool may not match the same number in another.

Two network evaluations per step has a cost worth keeping in mind. A 30-step sampler with guidance means about 60 network passes. A 1,000-step sampler with guidance would mean about 2,000.

Latent diffusion: why Stable Diffusion runs on a home GPU

Running diffusion directly on pixels is expensive. A 512 × 512 color image has 512 × 512 × 3 = 786,432 values, and each denoising step is a pass through a large network. The abstract of the latent diffusion paper notes that training pixel-space models can take hundreds of GPU days.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer proposed a workaround in December 2021: do the diffusion inside the latent space of a pretrained autoencoder. An autoencoder is a pair of networks. The encoder squeezes an image into a much smaller grid of numbers (the latent), and the decoder reconstructs the image from it. Think of it as zipping a photo, running the expensive work on the zip file, then unzipping at the end.

The numbers

Stable Diffusion's autoencoder uses a downsampling factor of 8, according to the CompVis README. For a 512 × 512 image, that gives a latent of 64 × 64 positions with 4 channels. The comparison:

	Pixel space	Latent space (Stable Diffusion v1)
Grid	512 × 512	64 × 64
Channels	3	4
Total values	786,432	16,384
Positions for attention to compare	262,144	4,096

The ratio of total values is 786,432 to 16,384, which is exactly 48. The count of positions drops by a factor of 64, and because self-attention compares every position with every other, a hypothetical full-resolution attention layer would have about 4,000 times more pairs to score. These figures are arithmetic from the published dimensions, not benchmark results.

From research code to a home computer

The base model in Stable Diffusion v1 has a 860 million parameter U-Net and a frozen 123 million parameter text encoder, per the CompVis repository. It was pretrained at 256 × 256 on a subset of LAION-5B and then fine-tuned at 512 × 512. LAION-5B, described in a 2022 paper by Christoph Schuhmann and colleagues, contains 5.85 billion CLIP-filtered image-text pairs, of which 2.32 billion are in English.

Stability AI's public release post is dated August 22 (2022). It states the memory use at release should be about 6.9 GB of VRAM, and the CompVis README says the reference code runs on a GPU with at least 10 GB. The model was released under a Creative ML OpenRAIL-M license, which Stability described as permitting commercial and non-commercial use while putting responsibility for ethical and legal use on the user. That combination of modest hardware needs and downloadable weights is why Stable Diffusion became the base for a wave of community tools, while DALL-E and Midjourney stayed hosted services.

The full text-to-image pipeline

Putting the pieces together, a Stable Diffusion v1 generation runs like this:

1. Encode the prompt. The text encoder turns the prompt into a sequence of embeddings.
2. Start from noise. Sample random Gaussian noise in latent space (64 × 64 × 4 for a 512 × 512 output).
3. Denoise iteratively. At each step, the U-Net predicts the noise, conditioned on the text through cross-attention and run with and without the prompt for guidance.
4. Update the latent. A sampler (DDPM, DDIM, or another) uses the prediction to compute the next, slightly cleaner latent.
5. Decode. After the final step, the autoencoder's decoder converts the clean latent into a full-resolution image.

If you want to see how this compares with the rest of an AI image product, including the interface and prompt handling, our overview of AI image generators covers the user-facing side, and how AI image generators understand prompts goes deeper on the text path.

Why diffusion models displaced GANs

Before 2021, generative adversarial networks (GANs) were the dominant way to generate images. A GAN pairs a generator with a discriminator that tries to catch fakes, and the two improve by competing. Our article on how GANs work explains the setup. Diffusion models overtook them for a mix of measurable and practical reasons.

On measurable quality, the numbers are specific. The DDPM paper's CIFAR-10 table lists a diffusion FID of 3.17 for unconditional generation, against 3.26 for StyleGAN2 with adaptive augmentation. That is a narrow win, and the same table shows a class-conditional StyleGAN2 variant at 2.67, so GANs were not simply beaten overnight. FID (Fréchet Inception Distance) is a score that compares statistics of generated and real images, where lower is better.

The stronger evidence came a year later. Prafulla Dhariwal and Alex Nichol's 2021 paper reports FID scores on ImageNet of 2.97 at 128 × 128, 4.59 at 256 × 256, and 7.72 at 512 × 512, and says that with as few as 25 forward passes per sample their models match BigGAN-deep while offering better coverage of the distribution.

The practical reasons, drawn from how the methods work rather than from a single benchmark, are these:

• A simple objective. Predict the noise and minimize squared error. There is no adversarial game to keep balanced.
• Natural conditioning. Cross-attention and guidance slot into the denoising network without redesigning the training setup.
• Coverage. The Dhariwal and Nichol abstract reports that, at matched quality, diffusion models kept better coverage of the data distribution than BigGAN-deep.

GANs keep one advantage, which is speed: one forward pass instead of many. That gap is what the next section is about.

Making diffusion fast

A thousand network passes per image is too slow for most uses, and a lot of research since 2020 has gone into cutting it. The table summarizes the main approaches, with figures taken from each paper's abstract.

Approach	Idea	Reported result
DDIM (Song, Meng, Ermon, 2020)	Non-Markovian sampler that skips steps	10 to 50 times faster wall-clock time than DDPM
Improved architecture (Dhariwal and Nichol, 2021)	Architecture changes found by ablation, sampled with far fewer steps	Matches BigGAN-deep with as few as 25 forward passes per sample
Consistency models (Song, Dhariwal, Chen, Sutskever, 2023)	Train the network to map any point on the noise path directly to the clean image	One-step FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64 × 64
Flow matching (Lipman et al., 2022)	Train on straight-line paths between noise and data	Reported better likelihood and sample quality than diffusion-based methods on ImageNet
Rectified flow transformers (Esser et al., 2024)	Connect noise and data in a straight line, with a transformer backbone	Outperforms established diffusion formulations in the authors' tests

These are self-reported results from the papers themselves, measured on research benchmarks. They do not tell you how fast a particular app will run on a particular phone.

What guidance, ControlNet, and recaptioning add

Three later additions change how people actually use diffusion models.

ControlNet. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala's 2023 paper adds spatial control to a pretrained text-to-image model. It keeps the original model frozen and trains a copy that learns extra conditions such as edge maps, depth maps, segmentation maps, and human pose, connected through layers initialized at zero so they do not disturb the original model at the start of training. You can hand it a stick-figure pose and get a picture of a person in that pose. The authors report that training works on both small (under 50,000 images) and large (over one million images) datasets.

Negative prompts. In common implementations, the "unconditional" branch of classifier-free guidance can be replaced by a second prompt describing what you do not want. The sampler then amplifies the difference between the positive prompt and the negative one, steering away from the unwanted content.

Better captions. OpenAI's DALL-E 3 paper, "Improving Image Generation with Better Captions," argues that prompt-following problems stem partly from noisy, short captions in web training data. The authors train a custom image captioner, use it to rewrite the training captions with richer descriptions, and report that training on these synthetic captions improves how well the model follows prompts. DALL-E 3 is a hosted service rather than a downloadable model, so outside researchers cannot inspect its weights.

Beyond still images

The same recipe extends to other kinds of data. OpenAI's Sora technical report (February 15, 2024) describes training text-conditioned diffusion models on videos and images of varying durations, resolutions, and aspect ratios, using a transformer on spacetime patches of latent codes. The report says the largest model can generate about a minute of video, and it states its own limits: it focuses on a unified representation and a qualitative look at capabilities, and does not give model and implementation details. Our explainer on what Sora is covers the product side.

The harder problems in video are consistency across frames and plausible motion, and they add heavy compute costs. The underlying framework is still noise in, structure out.

What diffusion models get wrong

Diffusion models make mistakes that follow from how they are built, and understanding them helps separate real limits from folklore.

They do not store a library of pictures. Stability's own release post describes the model as compressing visual information into "a few gigabytes," while the training data runs to billions of images. The weights hold statistical patterns, not a searchable archive.

They can still memorize some training images. A 2023 paper by Nicholas Carlini and colleagues, "Extracting Training Data from Diffusion Models," reports extracting over a thousand training examples from state-of-the-art models, ranging from photos of individual people to trademarked logos. The authors conclude that diffusion models are much less private than earlier generative models such as GANs. Memorization is rare relative to the volume of output, but it is not zero, and it matters for privacy and copyright debates. Our piece on the rise of AI art and the copyright controversy discusses that fight in more detail.

They inherit their data. LAION-5B was assembled by filtering web image-text pairs with CLIP. A dataset gathered that way reflects what is online, including its biases and its objectionable material, and filters can only catch part of it. This is why hosted services add output filters and open models have safety checkers that users can sometimes remove.

They struggle with details that need exact counting or exact spelling. Fingers, clocks, and printed words are common trouble spots. The causes are not fully settled. The Imagen paper found that a stronger text encoder improved alignment between prompt and image, and the DALL-E 3 paper ties prompt-following failures largely to caption quality, so better text understanding and better training captions both help, though neither removes the problem.

For the related question of how confident-sounding but wrong output arises in language models, see why AI hallucinates facts. The failure modes differ, but the shared theme is that a model trained to produce plausible output has no built-in way to check it against reality.

The takeaway

Diffusion models turned image generation from a delicate balancing act into a steady grind of small, learnable corrections. Training is easy to state: corrupt an image, ask the network what noise was added, and repeat for millions of images at random noise levels. Generation is the same skill played in reverse from static. Everything else in the field, from text conditioning to latent space to faster samplers, is engineering that makes this loop controllable, cheaper, and quicker. The most telling detail is how small the core idea is. A 2015 physics-inspired paper and a squared-error loss on predicted noise turned out to be enough to support a generation of image, and now video, models.

Key takeaways

• Diffusion models learn to reverse a noise-adding process: train on corrupted images, learn to predict the noise, then generate by starting from pure noise and removing it step by step.
• The original DDPM uses T = 1,000 steps with noise added at a rate rising linearly from 0.0001 to 0.02, ending in statistically pure noise.
• A U-Net with skip connections and self-attention was the standard denoiser; transformer backbones such as DiT now compete with and replace it.
• Text conditioning differs by system: DALL-E 2 predicts a CLIP image embedding, Imagen uses a T5 text encoder, and Stable Diffusion v1 uses a frozen CLIP text encoder with cross-attention.
• Classifier-free guidance runs the network with and without the prompt and exaggerates the difference, trading diversity for prompt adherence.
• Latent diffusion works on a 64 × 64 × 4 grid instead of 512 × 512 × 3 pixels, which is 48 times fewer values.
• Faster samplers (DDIM, consistency models, flow-based methods) cut the step count from 1,000 to as few as one in research settings.

FAQ

Q: Why does Stable Diffusion sometimes generate extra fingers? A: Hands are small, flexible, and appear in many poses, which makes their exact structure hard to learn. The causes are not fully settled. Better text encoders and better training captions have both been shown to improve prompt alignment, but neither removes the problem. Inpainting, which regenerates one region, and ControlNet pose conditioning are common workarounds.

Q: What is ControlNet? A: ControlNet is a method, published by Lvmin Zhang, Anyi Rao, and Maneesh Agrawala in 2023, that adds spatial conditions to a pretrained text-to-image diffusion model. It keeps the original model locked, trains a copy on conditions such as edges, depth, segmentation, or human pose, and joins the two through zero-initialized layers. The result follows both your text prompt and your structural guide.

Q: How is DALL-E 3 different from Stable Diffusion? A: DALL-E 3 is built around better training captions: OpenAI trained a custom captioner and used it to rewrite the training set, and reports improved prompt following. It is a hosted service. Stable Diffusion v1 was released with downloadable weights under a Creative ML OpenRAIL-M license, so people can run and fine-tune it locally. Both generate images by iterative denoising, but they differ in data, text conditioning, and access.

Q: What are negative prompts? A: A negative prompt is text describing what you want the image to avoid. In common implementations, it replaces the unconditional prediction in classifier-free guidance, so the sampler amplifies the difference between the positive and negative prompts. Adding "blurry" as a negative prompt pushes each step away from blurry-looking images.

Q: Can diffusion models generate video? A: Yes. OpenAI's Sora technical report describes text-conditioned diffusion models trained on videos and images, using a transformer on spacetime patches of latent codes, with the largest model able to generate about a minute of video. Video adds the problems of keeping objects consistent across frames and producing believable motion, and it raises compute needs considerably.

Q: Do diffusion models copy their training images? A: Mostly no, but not never. The model weights are a few gigabytes while the training data runs to billions of images, so the model stores patterns, not a library. Still, a 2023 study by Nicholas Carlini and colleagues extracted over a thousand training images from state-of-the-art models, including photos of individuals and trademarked logos. Memorization is a documented, measurable risk.

Q: Why does it take so many steps? A: Each step removes only a small amount of noise, because small corrections are what the network can learn reliably. The original DDPM used 1,000 steps. Later samplers such as DDIM report 10 to 50 times faster sampling, and consistency models report single-step generation, usually with some trade-off in quality that depends on the model and setting.

---

Read next:

• How Large Language Models Work: Transformers, Attention, and Why They Hallucinate
• Generative AI Art: How Diffusion Models Actually Work
• How Do Neural Networks Work? The Architecture Behind Modern AI
• Edge AI: How AI Runs on Your Phone
• What Are Deepfakes and How Do They Actually Work?

Sources

• Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015), Deep Unsupervised Learning using Nonequilibrium Thermodynamics, arXiv:1503.03585.
• Song, Y., and Ermon, S. (2019), Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019.
• Ho, J., Jain, A., and Abbeel, P. (2020), Denoising Diffusion Probabilistic Models, arXiv:2006.11239 (schedule, architecture, and loss details from the full paper).
• Song, J., Meng, C., and Ermon, S. (2020), Denoising Diffusion Implicit Models, ICLR 2021.
• Dhariwal, P., and Nichol, A. (2021), Diffusion Models Beat GANs on Image Synthesis.
• Radford, A., et al. (2021), Learning Transferable Visual Models From Natural Language Supervision (CLIP).
• Ho, J., and Salimans, T. (2022), Classifier-Free Diffusion Guidance.
• Rombach, R., et al. (2022), High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022.
• Ramesh, A., et al. (2022), Hierarchical Text-Conditional Image Generation with CLIP Latents.
• Saharia, C., et al. (2022), Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen).
• Schuhmann, C., et al. (2022), LAION-5B.
• Lipman, Y., et al. (2022), Flow Matching for Generative Modeling.
• Peebles, W., and Xie, S. (2022), Scalable Diffusion Models with Transformers.
• Carlini, N., et al. (2023), Extracting Training Data from Diffusion Models.
• Zhang, L., Rao, A., and Agrawala, M. (2023), Adding Conditional Control to Text-to-Image Diffusion Models.
• Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. (2023), Consistency Models.
• Esser, P., et al. (2024), Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.
• Ronneberger, O., Fischer, P., and Brox, T. (2015), U-Net: Convolutional Networks for Biomedical Image Segmentation.
• CompVis, stable-diffusion repository README.
• Stability AI, Stable Diffusion public release announcement.
• OpenAI (February 15, 2024), Video generation models as world simulators.