Crumb & Co. is adding a new loaf to the menu, and the owner wants a picture of it for the website. Instead of booking a photographer, she types a golden sourdough loaf on a wooden table, morning light into an image generator. A few seconds later, there's the picture:
Hitting the button again, with the exact same prompt, gives a completely different loaf:
Earlier in this series, we saw that a language model writes text one token at a time. A picture isn't made of words, though, and a video is a whole stack of pictures that have to agree with each other. So how does a model end up drawing a loaf? As it turns out, most of what we already know carries over, with a couple of new ideas added on top. We'll start with a quick review of how text gets written and build up from there.
Onwards!
A Quick Review: How Text Gets Written
Let's rewind to how Biscuit, the bakery's help desk bot, writes a reply. The text so far gets chopped into tokens, and the model gives every token in its vocabulary a score for how likely it is to come next. Those scores turn into odds, and the model picks one token from those odds. How adventurous that pick is comes down to temperature. The picked token gets added to the end, and the model runs again to get the next one. (Real systems save their work on the earlier tokens instead of redoing it, but every new token still depends on all of them.) Here is Biscuit finishing off the reply Our sourdough is unbelievably good!, one pass at a time:
Each pass adds exactly one token, and each pass has to wait for the one before it. Generating this way, where every new piece depends on all of the pieces before it, is called autoregressive generation.
Hang on to the big ideas here. The data becomes tokens, the model learns the odds of what comes next from an enormous pile of examples (as we saw in What Exactly Is an LLM?), and generating is a loop. Every image and video model we'll look at keeps some version of these.
Turning a Picture Into Tokens
To a computer, a picture is already a pile of numbers. Each pixel has a red, a green, and a blue value, so a 512 × 512 picture is 512 × 512 × 3 = 786,432 numbers. Treating every one of those numbers as its own token would turn even a small picture into a very VERY long sequence.
The fix is the one tokenizers use for text: work in chunks. We cut the picture into a grid of small squares called patches. Google's 2020 Vision Transformer paper made this popular, and its title says it all: "An Image is Worth 16x16 Words". It cuts a 224 × 224 picture into patches that are 16 pixels on a side, which works out to a 14 × 14 grid, or 196 patches. Each patch gets turned straight into a list of numbers for the model to read, the same way a text token becomes a list of numbers inside an LLM. Some image generators go one step further and swap each piece of the picture for a token ID from a fixed vocabulary, exactly like text. Here is that idea applied to our loaf, with a much coarser 4 × 4 grid so we can actually see what's going on:
The token IDs in that figure are made up, but OpenAI's original DALL-E, from 2021, trained a separate model to squeeze each 256 × 256 picture into a 32 × 32 grid of tokens, where every token is one of 8,192 possible codes. That list of codes is a vocabulary, just like a text tokenizer's, except each entry stands for a little patch of picture instead of a piece of a word.
Painting One Token at a Time
Once a picture is a sequence of tokens, we can generate it exactly the way we generate text. DALL-E put a caption of up to 256 text tokens first, followed by the picture's 1,024 image tokens, and trained a 12-billion-parameter model on 250 million captioned pictures to guess the next token. To make a new picture, we hand it only the caption. It then predicts image tokens one at a time, left to right and top to bottom:
When the last token is in, a decoder turns the grid of codes back into pixels. Google's 20-billion-parameter Parti works this way, and so does a model many of us have used. When OpenAI added image generation to GPT-4o in 2025, its system card addendum put it like this:
Unlike DALL·E, which operates as a diffusion model, 4o image generation is an autoregressive model natively embedded within ChatGPT
This approach has the same weakness text does. A token can't be taken back once it's picked, so if the top of the loaf lands a little too far left, every patch after it has to live with that. Notice the word diffusion in that quote, too. It's the other big family of image models.
Starting From Noise: Diffusion
Instead of building a picture one piece at a time, a diffusion model starts with a picture made entirely of random static. It then cleans that static up a little bit at a time, until a loaf comes out of the fuzz.
Training works like this. We take a real photo and add a small amount of random noise to it. Then we add a little more, and a little more, until nothing is left but static. The authors of the 2020 DDPM paper used 1,000 of these small steps. Because we added the noise ourselves, we know exactly what it was. This means we can ask the model a question with a known answer: looking at this noisy picture, what noise was added? After millions of pictures, it gets very good at spotting noise. To generate, we run the process in reverse:
We start from pure static, ask the model what noise it sees, use that guess to take a small step toward a cleaner picture, and repeat. (That's the intuition. The real math involves a sampler that decides how big each step is, but the idea holds.) Roughly speaking, the early steps settle the rough shapes and colors, and the later steps fill in the crust and the crumb. Every step works on the whole picture at once, so a mistake in one corner can still get fixed on a later step instead of being locked in the way an already-picked token is.
The starting static is random, which is why the same prompt gives a different loaf every time, much like sampling does for text. The price we pay is that every step is a full trip through the model. Stable Diffusion's original sample script, for example, uses 50 steps by default.
How the Words Steer the Picture
So far, our diffusion model can turn static into some picture. How does it know we asked for a sourdough loaf and not a cat? It needs a way to connect words to pictures, and that's where embeddings come in.
An embedding is a list of numbers that captures what something means, arranged so that similar things get similar lists. If we picture each list as a point in space, similar things land close to each other. (Embeddings are also the heart of searching text by meaning, which the Embeddings article covers in depth.) OpenAI's CLIP, from 2021, learned embeddings for pictures and captions in one shared space. It trained on 400 million picture-and-caption pairs collected from the internet, and the goal was simple: pull each picture's embedding close to its own caption's embedding, and push it away from everyone else's. Here is what that looks like, squashed down from hundreds of dimensions to two:
A text encoder trained this way turns our prompt into numbers that already know something about what pictures look like. Stable Diffusion, for instance, uses a frozen copy of CLIP's text encoder (its weights stay fixed while the diffusion model trains) and hands its output to every single denoising step. Because the model trained on noisy pictures paired with their captions, it learns to remove noise in whichever direction makes the picture match the caption.
There's one more knob. A technique called classifier-free guidance asks the model for two predictions at every step, one with our prompt and one with no prompt at all. The difference between those two predictions points, roughly, toward our prompt, and the model pushes the picture further in that direction. Turning the guidance up makes the picture follow the prompt more closely, but there's a catch, and Google's Imagen paper spells it out:
Increasing the classifier-free guidance weight improves image-text alignment, but damages image fidelity producing highly saturated and unnatural images
We'll run into that oversaturated look again in the next article.
Working in a Smaller Space
Running 50 steps over 786,432 numbers is a lot of work, and plenty of those numbers describe fine detail our eyes barely notice. The 2022 latent diffusion paper, the research behind Stable Diffusion, fixed this by compressing first. A separate model called an autoencoder learns to squeeze a picture into a much smaller grid of numbers and to expand it back into pixels with very little visible loss. That small grid is called the latent, and the space of all possible latents is called latent space. For the first version of Stable Diffusion, the numbers look like this:
That's 48 times fewer numbers to clean up on every step, and the decoder runs just once at the very end. Video models lean on this same trick, which is a good thing, because video needs all the help it can get.
Video Is Pictures Over Time
A video is a stack of pictures, called frames, shown quickly one after another. A 16-second clip at 16 frames per second is 256 frames. If we generated each frame separately, the loaf might change shape from one frame to the next, and the video would flicker like a bad dream. What makes video tricky is keeping the same loaf, on the same table, moving in a way that makes physical sense.
OpenAI's technical report on Sora describes one way to handle this. First, the video gets compressed into a latent that is smaller in both space and time. Then that latent is cut into spacetime patches, chunks that cover a small area of the picture across several frames. Here is the idea on a tiny four-frame clip of our loaf rolling across the table:
OpenAI hasn't published Sora's actual patch sizes, so the two-frame patches in that figure are just to show the idea. Each spacetime patch becomes a token, and Sora is a diffusion model built on the same transformer design that LLMs use. So basically, it starts with a whole clip's worth of noisy tokens and denoises them together, and the transformer can relate any patch to any other, across both space and time. OpenAI's Sora announcement says that giving the model "foresight of many frames at a time" is how they made sure "a subject stays the same even when it goes out of view temporarily".
The numbers get big fast. Meta's Movie Gen paper describes a 30-billion-parameter video model that works on up to 73,000 video tokens at once, which is what a 16-second clip at 16 frames per second takes. And consistency is still hard. The same Sora announcement admits as much:
It may struggle to simulate the physics of a complex scene, and may not comprehend specific instances of cause and effect (for example: a cookie might not show a mark after a character bites it).
Where the Training Pictures Come From
The recipe is the same one we saw for text, an enormous pile of examples, except that for a text-to-image model every picture needs a caption so the model can learn which words go with which pictures. LAION-5B, the open dataset that Stable Diffusion's training pictures came from, has 5.85 billion picture-and-text pairs, pulled from images and their alt text across the web and kept only when CLIP agreed the two matched. Alt text is often vague, like IMG_2041, so for the DALL-E 3 paper, OpenAI trained a separate model to write detailed captions and trained mostly on those, which made the image model much better at following prompts. The Sora announcement says Sora uses this same recaptioning technique for video.
Why Pictures Cost So Much More
We now have enough pieces to see why pictures and video take so much more work than a chat reply. Here is some very rough counting of how many pieces (tokens or latent positions) the model handles for each:
- Biscuit's 100-token reply takes 100 passes, and each pass produces one new token.
- A Stable Diffusion picture takes 50 steps over a 64 × 64 latent, which is 4,096 positions per step (each holding 4 numbers), or about 200,000 in total.
- A 16-second Movie Gen clip has up to 73,000 tokens per step. If it took the same 50 steps (our assumption, to keep the math simple), that's about 3.65 million.
Here are those three counts drawn on the same scale:
These are counts of how much data the model handles, not exact amounts of math, since the models differ a lot in size and design. They still show the shape of the problem. A picture is far more data than a sentence, a video is far more again, and diffusion goes over that data many times. All of that work runs on the same kind of GPU hardware that trains these models in the first place, which is a story we'll get to in Why Training Takes a Data Center.
Conclusion
When Crumb & Co.'s owner typed her sentence and got back a golden loaf, the model was doing something we already understood from text. Autoregressive image models write the picture out token by token, the way Biscuit writes a reply. Diffusion models start from static and remove noise step by step, with an embedding of the prompt steering every step and a compressed latent keeping the work manageable. Video adds time on top, with spacetime patches helping the loaf stay the same loaf from frame to frame.
Under the hood, every picture, latent, and video clip in this article was a block of numbers with a very particular shape. Those blocks are called tensors, and Ben Affleck recently explained them in a single sentence that's worth unpacking slowly. That's where we're going next in What Is a Tensor?
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
