# How Transformers and Attention Work by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm) A customer types this into the chat box on the Crumb & Co. bakery's website, and **Biscuit**, our help desk bot, answers right away: ![The Biscuit chat window on the Crumb & Co. bakery website. A customer asks, Mom loved the sourdough. Is it vegan? Biscuit replies, Glad she loved it! Yes, our sourdough is vegan.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_chat_it.png) Look at the little word ***it***. You knew instantly that ***it*** means the sourdough and not Mom, and so did Biscuit. For a model, that's harder than it looks. In [What Is a Tensor?](https://www.kirupa.com/ai_fundamentals/what_is_a_tensor.htm), we saw that every token becomes a list of numbers from the model's embedding table. The list for ***it*** is the same in every sentence, so on its own, it has no idea what this particular ***it*** points to. The model has to work that out from the tokens around it. The part of the model that does this is called **attention**. It's the heart of the **transformer**, the design Ashish Vaswani and seven colleagues at Google introduced in their 2017 paper ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762), and nearly every LLM in this series is a transformer. In this article, we'll see why older models struggled, work through attention by hand, and build up to a full transformer, including why it loves GPUs and why long conversations get expensive. Onwards! ## Reading One Word at a Time Before transformers, the go-to design for text was the **recurrent neural network** (aka RNN). An RNN reads one token at a time, in order, while carrying a running summary, which is a fixed-size list of numbers. At every step, it mixes the next token into the summary and hands the result to the next step. Here is an RNN reading our customer's message, with each word treated as one token to keep things small (a real tokenizer might chop up ***sourdough***, as we saw in [Tokens and How They Set Your Bill](https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm)): ![A recurrent network reads the message Mom loved the sourdough. Is it, one token per step, from step 1 to step 7. Each token feeds into a running summary of the same small size, and each step hands its summary down to the next. By step 7, when it arrives, everything about the earlier tokens has to fit in that same small box.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_rnn.png) Follow the summary from top to bottom. By the time the RNN reaches ***it***, everything it knows about the earlier tokens has to fit in one small box that has been rewritten six times. That makes an RNN slow, and it makes it forgetful. It's slow because step 7 can't start until step 6 is done, and the transformer paper put it bluntly: an RNN's ***"inherently sequential nature precludes parallelization within training examples"***. In plain words, an RNN can't spread one piece of text across a GPU's thousands of processors. It's forgetful because the summary never gets any bigger, so details from the start of a long message tend to fade by the end. The best-known fix, the [Long Short-Term Memory](https://doi.org/10.1162/neco.1997.9.8.1735) design (LSTM) from 1997, helped networks hold onto information for much longer, but they still read one token at a time. ## Attention: Looking Back at Everything The transformer's big idea is to drop the running summary. Instead, each token looks directly at every token before it and decides which ones matter. Here is ***it*** doing exactly that, with a thicker line for each token it pays more attention to: ![The message Mom loved the sourdough. Is it vegan? as a row of tokens. Lines arc from it back to every earlier token: Mom, loved, the, sourdough, the period, and Is. The line to sourdough is by far the thickest, because it pays the most attention to sourdough. The two tokens after it, vegan and the question mark, are grayed out.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_attention.png) Look at the thickest line. ***it*** reaches straight back to ***sourdough*** in a single hop, and it could do the same if the sourdough were a thousand tokens back, so nothing fades along the way. (We'll get to the two grayed-out tokens shortly.) The original paper even caught this happening. Its authors found attention heads in their trained model that were ***"apparently involved in anaphora resolution"***, which is a fancy name for working out what a word like ***it*** points back to. So how does a token decide which earlier tokens matter? That's where queries, keys, and values come in. ## Queries, Keys, and Values Let's step into the pantry at Crumb & Co. Every jar has a label on the front and an ingredient inside. A baker walks in holding a note that says what they need, compares the note against the labels, and scoops from the jars whose labels match best. Attention works the same way, and every token carries all three things: - A **query** is the note, which describes what a token is looking for. For ***it***, that's roughly ***an earlier thing I could be standing in for***. - A **key** is the label, which describes what a token has to offer. The key for ***sourdough*** says something like ***a thing you can eat***. - A **value** is the ingredient in the jar, which is the information a token hands over when it gets picked. Each layer has three weight matrices, one each for queries, keys, and values. Multiplying a token's numbers by each matrix, the same way we multiplied by hand in What Is a Tensor?, gives that token its query, key, and value. Here is ***it***, shrunk down to three numbers: ![The token it has the numbers 1, 2, and 0. Multiplying them by the query weights gives its query, 1 and 2, which describes what it is looking for. Multiplying them by the key weights gives its key, 1 and 1, which describes what it has to offer. Multiplying them by the value weights gives its value, 5 and 5, which is what it hands over.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_qkv.png) Each number in a result comes from walking down one column of a matrix, so the first number of the query is 1 × 1 + 2 × 0 + 0 × 2 = 1. Every token gets its query, key, and value from these same three matrices, and the model learns the numbers inside them during training. Our notes and labels are just a way to picture it. In a real model, they're lists of numbers with no tidy names. ## Attention by Hand Now let's run attention for ***it***. To keep the numbers friendly, we'll look at three of the tokens it can see: ***Mom***, ***sourdough***, and ***it*** itself, since a token can look at itself too. Step 1 scores each token's key against the query for ***it*** using a **dot product**: multiply the matching numbers and add up the results. Step 2 turns the scores into shares that add up to 100% with a function called **softmax**, which raises the number e (about 2.718) to the power of each score and divides each result by the total: ![The query for it is 1 and 2. Step 1 scores each key with a dot product: Mom's key, 2 and 0, scores 1 times 2 plus 2 times 0, which is 2. Sourdough's key, 1 and 2, scores 1 times 1 plus 2 times 2, which is 5. The key for it, 1 and 1, scores 3. Step 2 raises e to each score, giving about 7.4, 148.4, and 20.1, which add up to about 175.9, and divides each by that total. Mom gets a 4.2 percent share, sourdough gets 84.4 percent, and it gets 11.4 percent.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_scores.png) Look at the ***sourdough*** row. Its key points the same way as the query, so it gets the biggest score, 5. Softmax then stretches that lead. A 5 is only 2 more than a 3, but e to the 5th power is more than seven times e to the 3rd, so ***sourdough*** ends up with 84.4% of the attention. Step 3 blends the values. Each token's value gets multiplied by its share, and the results get added together: ![Step 3 blends the values. Mom's share of 4.2 percent times its value, 10 and 0, gives 0.42 and 0. Sourdough's share of 84.4 percent times its value, 0 and 10, gives 0 and 8.44. The share for it, 11.4 percent, times its value, 5 and 5, gives 0.57 and 0.57. Adding them up gives 0.99 and 9.01, the new numbers for it.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_mix.png) The new numbers for ***it***, [0.99, 9.01], are mostly the value from ***sourdough***, with a little of the other two mixed in. If we pretend the first number in each value means ***person*** and the second means ***bread***, then ***it*** now leans heavily toward bread, which later tokens like ***vegan*** can use. Real models do the same thing with every earlier token and with much longer queries and keys. They also divide each score by the square root of the key's length before the softmax. That keeps long vectors from producing scores so large that training stalls, and it's why the paper calls this ***scaled dot-product attention***. With every token's query, key, and value stacked into matrices called Q, K, and V, the whole calculation fits on one line: ```text Attention(Q, K, V) = softmax(Q × Kᵀ / √d) × V ``` Kᵀ is K turned on its side (its **transpose**), so one matrix multiplication scores every query against every key at once. ## No Peeking Ahead When ***it*** looked around, it only looked back. It never looked ahead at ***vegan*** or the question mark, even though they're right there in the message. Why? As we saw in [What Exactly Is an LLM?](https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), training is a guessing game where the model predicts each next token. If a token could peek at the token it's supposed to guess, the model would just copy the answer. So every transformer that writes text blocks each token from seeing the tokens after it, which is called **causal masking**. Here is the mask for our message, with a row for each token doing the looking and a column for each token being looked at: ![A 9 by 9 grid for the message Mom loved the sourdough. Is it vegan? Each row is a token doing the looking, and each column is a token being looked at. Every row can look at the cells up to and including itself, and every cell after that is blocked, so the grid is filled in below its diagonal. The row for it is outlined: it can look at Mom through it, while vegan and the question mark are blocked. A handwritten note says: no peeking!](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_mask.png) Find the row for ***it***. It can use every cell up to and including itself, and the two cells after it are blocked. The model blocks them by setting their scores to minus infinity before the softmax. Since e to the power of minus infinity is 0, those tokens get a 0% share. ## Many Heads at Once A token often needs to find several things at once. What does it stand for? What came right before it? Where does the message start? One set of queries, keys, and values blends everything in one particular way, so a transformer runs several attention calculations side by side, each with its own three weight matrices. Each one is called an **attention head**, and here are three of them looking out from ***it***: ![Three attention heads look out from it in the same message. Head 1, what does it stand for?, draws its thickest line to sourdough. Head 2, what came right before?, draws its thickest line to Is. Head 3, where does the message start?, draws its thickest line to Mom. Each head has faint lines to the other earlier tokens too.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_heads.png) Head 1 pulls in the sourdough, head 2 pulls in ***Is***, the token right before, and head 3 pulls in ***Mom***, where the message starts. The model glues the heads' results back together and runs them through one more weight matrix. Nobody assigns those jobs, though. They're learned in training, and researchers work out afterward what each head seems to do. The original paper used 8 heads per layer and reported that ***"the heads clearly learned to perform different tasks"***. At the big end, Llama 3.1 405B runs 128 heads in each layer, and each head works with a slice of 128 of the 16,384 numbers every token carries. ## One Transformer Block Attention is only half of a layer. Here is one full layer, which is also called a **transformer block**, along with how many times it repeats: ![One transformer block, drawn top to bottom. Each token's numbers go into attention, where tokens look at each other, then into an add and normalize step. A residual line carries the numbers around attention and adds them back in at that step. Then comes feed-forward, where each token is worked on by itself, followed by a second add and normalize step with its own residual line around feed-forward. The result goes on to the next block. Beside it, a stack of blocks from block 1 up to block 126 shows that Llama 3.1 405B stacks 126 of them, with next-token scores coming out of the top.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_block.png) After attention comes the **feed-forward** part, which runs each token's numbers through a couple of big weight matrices, with each token on its own. Attention is where tokens gather information from each other, and feed-forward is where each token works with what it gathered. (In [Types of AI Models](https://www.kirupa.com/ai_fundamentals/types_of_ai_models.htm), we'll see that this part holds most of a model's parameters.) Each ***add & normalize*** step starts with a **residual connection**, which adds the part's output back onto the numbers that went into it. That way, attention adds to the numbers for ***it*** instead of replacing them, a bit like writing notes in the margin of a recipe card. Then the numbers get **normalized**, which means rescaling them to a steady size so they don't drift bigger or smaller as they pass through dozens of blocks. Together, they make tall stacks of blocks much easier to train. The original transformer was built for translation, with one stack of blocks to read and another to write. [OpenAI's first GPT](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf), in 2018, kept just one stack of masked blocks, and LLMs like Llama are built the same way. At the very top of the stack, the numbers for the last token become a score for every token in the vocabulary. That's the next-token guess from What Exactly Is an LLM?, and it's how Biscuit picks each token of its reply. ## Where Position Comes In There's a strange gap in all of this. Nothing in a dot product says how far apart two tokens are, so a ***sourdough*** two tokens back and a ***sourdough*** two hundred tokens back would get the same score. Word order matters, though. ***The baker bit the dog*** is a very different story from ***the dog bit the baker***. The original paper added a pattern of sine and cosine waves to each token's numbers, with a different pattern for every position, a bit like stamping each token with its seat number. Many open models today, including Llama 3, use a newer trick called [rotary position embedding](https://arxiv.org/abs/2104.09864) (aka RoPE), which rotates each token's query and key by an angle that grows with the token's position. Here is the idea with one pair of numbers turning 30° per position: ![Rotary position embedding with one pair of numbers turning 30 degrees per position. Above each token of the message is a dial whose hand has turned by its position: Mom 0 degrees, loved 30, the 60, sourdough 90, the period 120, Is 150, and it 180. Two pairs are bracketed: Mom and the sit 2 apart with a 60 degree turn between them, and sourdough and Is also sit 2 apart with the same 60 degree turn.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_position.png) ***Mom*** and ***the*** sit 2 positions apart, and their hands are 60° apart. ***sourdough*** and ***Is*** also sit 2 positions apart, and their hands are 60° apart too. The angle between two tokens depends only on how far apart they sit, and a dot product changes with that angle, so the attention score now knows the distance. Real models turn many pairs of numbers at different speeds, like the second, minute, and hour hands on a clock, so they can tell near from far at many scales. ## Why GPUs Love Transformers Now we can see why transformers took over. Look back at the mask grid. No row waits on another, so the model can play the guessing game at every position of a sentence in one pass. From our one message, it practices guessing ***loved*** from ***Mom***, ***the*** from ***Mom loved***, and so on, all at the same time. On a long piece of training text, those guesses become a handful of giant matrix multiplications. As we saw in What Is a Tensor?, that's exactly the kind of work a GPU's thousands of small processors are built for, while an RNN has to march through the same text one step at a time. The payoff showed up right away. The paper's model set a new best score for translation after training ***"for as little as twelve hours on eight P100 GPUs"***. That efficiency is what made it practical to train far bigger models on far more text, all the way up to the thousands of GPUs in [Why Training Takes a Data Center](https://www.kirupa.com/ai_fundamentals/data_centers_and_training.htm). Writing a reply is still one token at a time, though, as we saw in Tokens and How They Set Your Bill, because each new token depends on the one before it. ## Why Long Contexts Cost More Attention has a catch, and it's baked into the idea itself: every token looks at every earlier token. In a message of n tokens, the first token makes 1 comparison, the second makes 2, and so on, which adds up to about n² ÷ 2 comparisons for every head in every layer: ![Each square is one comparison between a token and an earlier token or itself. 4 tokens make a small triangle of 10 comparisons, and 8 tokens make a triangle of 36. A table continues the pattern: 1,000 tokens take about 500,000 comparisons, 10,000 tokens take about 50 million, and 100,000 tokens take about 5 billion, for every head in every layer.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_cost.png) Going from 4 tokens to 8 grows the comparisons from 10 to 36. At 1,000 tokens, it's about half a million, and at 100,000 tokens, it's about 5 billion. So basically, 10 times the text means about 100 times the attention work. That's one reason long inputs are slower, and part of why [Context Window Limits](https://www.kirupa.com/ai_fundamentals/context_window_limits.htm) suggested keeping the model's desk tidy. That one-at-a-time writing adds a memory catch. Every new token needs the keys and values of every token before it, and thanks to the mask, those never change. Working them out again for every new token would be a huge waste, so the model saves them as it goes, in what's called the **KV cache** (short for key-value cache). We can think of it as the model's notes on each token: ![The KV cache as a list of notes, one row per token. The customer's message, Mom loved the sourdough. Is it vegan?, and the reply so far, Glad she, each have a saved key and value, labeled saved notes, reused for every new token. The next reply token, loved, is in the last row with its own query, key, and value, labeled worked out now. Earlier tokens don't keep a query.](https://www.kirupa.com/ai_fundamentals/images/how_transformers_work_kv_cache.png) The new token in the last row works out only its own query, key, and value. Everything else comes from the saved notes. The trouble is that the notes grow with every token. Meta's published settings for Llama 3.1 405B give it 126 layers, and in each layer, its heads share 8 sets of keys and values with 128 numbers each. With one key and one value per set and 2 bytes per number, our own arithmetic is 126 × 8 × 128 × 2 × 2 = 516,096 bytes, or about 0.5 MB of notes per token. A 128,000-token conversation needs about 66 GB of notes on top of the weights. That's also why 128 heads share just 8 sets. [Meta's Llama 3 paper](https://arxiv.org/abs/2407.21783) says the trick, called grouped query attention, is there ***"to reduce the size of key-value caches during decoding"***. [Prompt Caching](https://www.kirupa.com/ai_fundamentals/prompt_caching.htm) shows how providers save these notes between requests, and [Time to First Token](https://www.kirupa.com/ai_fundamentals/time_to_first_token.htm) looks at the wait while the model reads our prompt and builds them. ## Going Further If you'd like another angle on all of this, Jay Alammar's ["The Illustrated Transformer"](https://jalammar.github.io/illustrated-transformer/) walks through the original design one picture at a time. Andrej Karpathy's ["Let's build GPT: from scratch, in code, spelled out."](https://www.youtube.com/watch?v=kCc8FmEb1nY) is a nearly two-hour video where he writes a small transformer in Python, line by line, and trains it on a file of Shakespeare. ## Conclusion We started with Biscuit figuring out that ***it*** meant the sourdough, and now we can see how. The query for ***it*** got scored against the keys of earlier tokens, softmax turned those scores into shares, and ***it*** took in a blend of their values. A mask keeps every token from peeking ahead, several heads look for different things at once, and position lets the model tell near from far. Feed-forward, residual connections, and normalization round out each block, and stacking that block dozens of times gives us a transformer. It trains beautifully on GPUs because every row of the mask can be computed at once, and long contexts get expensive because every token looks back at every earlier one. All of that machinery exists to produce one thing: the most plausible next token, given everything the model has seen. That's what makes these models so useful, and it's also why so much of what they produce ends up sounding the same. That sameness has a name, and it's where we're going next in [What Is AI Slop?](https://www.kirupa.com/ai_fundamentals/what_is_ai_slop.htm)