# Why Training Takes a Data Center

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

In [What Exactly Is an LLM?](https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), we saw that Meta trained the biggest Llama 3.1 model on up to 16,000 H100 GPUs. That's a lot of graphics cards. Plenty of us have a pretty capable graphics card at home, the kind that runs the latest games without breaking a sweat. So why can't one of those train a model, even if it takes a while? And why do the companies that train these models fill entire buildings with GPUs and keep them running day and night for months?

It all comes down to arithmetic, and the numbers are big enough that working through them is kind of fun. We'll use Llama 3.1 405B as our running example the whole way, because Meta published an unusually detailed account of how they trained it in [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783). We'll stick to published figures, and when a number is our own estimate, we'll say so.

Onwards!

## Counting the Math

Let's start by counting how much math training takes. Training shows the model some text, has it guess each next token, checks those guesses against what actually came next, and nudges every weight in the direction that would have made the right answers more likely. That happens in two parts:

- The **forward pass** runs the text through the model to get its guesses. Every parameter takes part in about one multiplication and one addition per token, so that's roughly 2 operations per parameter, per token.
- The **backward pass** works out how much to nudge each weight, which is called the **gradient**. It takes about twice the math of the forward pass, so roughly 4 more operations per parameter, per token.

Add those up, and we get about 6 operations for every parameter, for every token the model trains on. This is a well-known rule of thumb. OpenAI's 2020 [scaling laws paper](https://arxiv.org/abs/2001.08361) estimated training compute as about 6N floating-point operations per training token, where N is the parameter count, and DeepMind's [Chinchilla paper](https://arxiv.org/abs/2203.15556) used the same 6ND shortcut, where D is the number of training tokens. (A **floating-point operation**, or FLOP, is a single arithmetic operation, like one multiplication or one addition, on the kind of numbers with a decimal point that models use.) Here is that shortcut applied to Llama 3.1 405B:

![The training math for Llama 3.1 405B: 6 operations per parameter per token, times 405 billion parameters, times 15.6 trillion training tokens, equals about 3.8 times 10 to the 25th floating-point operations, which is 38 followed by 24 zeros.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_six_nd.png)

Our back-of-the-envelope math lands right on the 3.8 × 10^25 FLOPs that Meta reports in its paper. That's 38 followed by 24 zeros, and it's a number that's hard to feel. So let's hand it to a graphics card and see what happens.

## One Graphics Card Against the Math

NVIDIA's GeForce RTX 4090 is a top-end gaming card. According to NVIDIA's [Ada architecture whitepaper](https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf), it peaks at 165.2 trillion operations per second for the kind of 16-bit math training uses. Here is how it stacks up against our training job:

![A single RTX 4090 running at its theoretical peak would need about 7,300 years to do the math for Llama 3.1 405B. Meta reported 30.84 million H100 GPU hours for the job, which works out to about 78 days if 16,384 GPUs ran the whole time.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_one_gpu.png)

Dividing 3.8 × 10^25 operations by 165.2 trillion per second gives about 230 billion seconds, which is roughly 7,300 years. And that's the generous version, since no real job runs a GPU at its peak speed. Meta reports that its H100s reached a **model FLOPs utilization** of 38% to 43%, meaning the model's math ran at that fraction of the GPUs' peak speed, with the rest lost to things like waiting on memory and on other GPUs. If our gaming card somehow managed 40% too (our own made-up scenario), it would need about 18,000 years.

The bottom number is our own estimate. The [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md) says training the 405B model took 30.84 million GPU hours on H100s. If all 16,384 GPUs had run the whole time, that would be about 78 days. Meta's paper shows some stages of training used 8,192 GPUs instead, so the real run took longer than that. Each [H100](https://www.nvidia.com/en-us/data-center/h100/) is also about six times faster than an RTX 4090 at this kind of math, with 989 trillion operations per second.

Here is one H100 in a [photo by 极客湾 Geekerwan](https://commons.wikimedia.org/wiki/File:NVIDIA_H100_(%E6%9E%81%E5%AE%A2%E6%B9%BEGeekerwan)_014.png) ([CC BY 3.0](https://creativecommons.org/licenses/by/3.0/), cropped). This is the plug-in card version. Meta's cluster used the [SXM version](https://www.nvidia.com/en-us/data-center/h100/), which mounts flat on a server board and is rated for up to 700 watts:

![An NVIDIA H100 graphics card in its PCIe form, a long gold-colored card with a metal bracket at one end and a gold edge connector along the bottom, standing on a white table.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_h100_photo.jpg)

Cards like this don't live in desktop towers. They live in data centers, so let's zoom in from a whole building down to a single GPU. When Meta expanded its Prineville campus, it [added 900,000 square feet with two new buildings](https://datacenters.atmeta.com/2021/03/facebooks-prineville-data-center-is-growing-again/), or about 450,000 square feet each. A rack built to the Open Compute Project's [Open Rack V3 design](https://www.opencompute.org/marketplace/facilities-physical-infrastructure/sanmina-open-rack-v3-orv3/) is 600 mm wide, 1,068 mm deep, and about 2.3 m tall. Here is one building that size, with one rack drawn at the same scale:

![An isometric drawing of one long, low data center building with rows of cooling units on its roof and a parking lot of cars beside it, drawn to scale for about 450,000 square feet of floor space on one level. In front of the building, a rack drawn at true scale is so small it is just a speck inside a red circle, with a handwritten note: one rack is about this big. An inset shows the same rack enlarged, standing next to a person for scale.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_dc3d_building.png)

Inside that building are halls full of racks. Here is a simplified slice of one, with one rack picked out:

![An isometric drawing of a slice of a data center hall: four long rows of identical dark gray racks standing on the floor, with aisles between the rows. The rack at the end of the front row is highlighted in blue with a handwritten note: this rack!](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_dc3d_hall.png)

In Meta's cluster, the [Llama 3 paper](https://arxiv.org/abs/2407.21783) says each rack held two servers and a network switch, and each server held eight GPUs. Rack layouts vary from one data center to the next, so think of this as one real example. Here is that rack opened up:

![An isometric drawing of one blue rack with its front open. A purple network switch sits near the top. Two server drawers are pulled out of the front, and each drawer has 8 gold H100 GPUs on top, for 16 H100s in the rack, matching the layout Meta describes for its Llama 3 cluster.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_dc3d_rack.png)

Each of those gold blocks is one H100 in its SXM version, the same chip as the card in the photo but mounted flat on the server board. Here is one up close:

![An isometric drawing of one H100 in its SXM version: a flat green circuit board with a large gold finned heatsink on top. Labels point to the heatsink on top, the GPU chip and its 80 GB of memory under the heatsink, and the circuit board. The module is rated for up to 700 watts.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_dc3d_gpu.png)

Meta trained Llama 3.1 405B on up to 16,384 of these GPUs, which is 1,024 of those 16-GPU racks. Here is what that many looks like, next to the single GPU in a gaming PC:

![A 32 by 32 grid of 1,024 small squares, where each square is one rack of 16 H100 GPUs, for 16,384 GPUs in total, the most Meta used to train Llama 3. One square near the top right is highlighted in red with a handwritten note: one rack! Beside the grid, one square is zoomed in to show its 16 GPUs as 2 servers of 8, and a single square stands for the one GPU in a gaming PC.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_16k.png)

So time is a problem. As it turns out, it isn't even the first one we'd hit.

## The Memory Wall

Even if we had 7,300 years to spare, the model wouldn't fit on the card. Training needs a lot more than the weights in memory at once:

- The **weights** themselves, stored at 2 bytes each. For 405 billion weights, that's 810 GB.
- The **gradients**, one for every weight, at another 2 bytes each.
- The **optimizer state**. The optimizer is the piece of the training code that decides exactly how to apply the gradients. The most common one, called Adam, keeps a high-precision copy of every weight plus two running averages per weight, at 4 bytes apiece.

Microsoft's [ZeRO paper](https://arxiv.org/abs/1910.02054) adds all of this up to 16 bytes for every parameter. For Llama 3.1 405B, that looks like this:

![Training Llama 3.1 405B with the Adam optimizer needs about 16 bytes per parameter: 810 GB for the weights, 810 GB for the gradients, and 4,860 GB for the optimizer state, about 6,480 GB before activations. At the same scale, an RTX 4090's 24 GB and an H100's 80 GB are tiny slivers.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_memory.png)

That's about 6,480 GB, and we're not done. The forward pass also produces in-between results, called **activations**, that have to be kept around for the backward pass, and those can add a lot more on top. An RTX 4090 has 24 GB of memory. That's about 270 times the card's memory. Even an H100, with 80 GB, would need 81 of itself just to hold the training state of this one model. So before speed even enters the picture, the work has to be spread across many GPUs just to fit.

## Splitting the Work

So how do thousands of GPUs train one model together? Meta's paper describes using four kinds of splitting at the same time, which it calls 4D parallelism. Three of them do most of the work, and here is what they look like with four GPUs each:

![Data parallel: four GPUs each hold a full copy of the model and train on different data, A through D. Tensor parallel: a single layer of the model is split into four pieces, one per GPU. Pipeline parallel: four GPUs each hold one quarter of the model's layers, and the work flows from GPU 1 to GPU 4 in order.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_parallelism.png)

**Data parallelism** gives each GPU (or group of GPUs) a copy of the model and a different slice of the training text. Each one works out gradients for its own slice, and then they all average their gradients together so every copy makes the exact same update and stays in sync. That basic version needs every copy to fit on its GPU, which our model can't. So Meta used a fully sharded version, where each GPU keeps only a slice of the weights, gradients, and optimizer state, and borrows the other slices from its neighbors for just as long as it needs them.

**Tensor parallelism** splits the giant grids of numbers inside each layer across several GPUs, an approach popularized by NVIDIA's [Megatron-LM paper](https://arxiv.org/abs/1909.08053). Each GPU does its share of a layer's math, and then they combine their pieces before moving on to the next layer.

**Pipeline parallelism** gives each GPU a consecutive chunk of the model's layers, like stations on an assembly line. To keep every station busy, the training text is cut into small micro-batches that follow each other down the line. Even so, the stations at the end sit idle while the line fills up, and the ones at the start sit idle while it drains. Google's [GPipe paper](https://arxiv.org/abs/1811.06965) calls that idle time the ***bubble***.

The fourth kind, **context parallelism**, splits very long pieces of text across GPUs so that a long [context window](https://www.kirupa.com/ai_fundamentals/context_window_limits.htm) doesn't overwhelm any one of them. Combining all four is how Meta fit a model that needs thousands of gigabytes of training state onto GPUs with 80 GB each.

## Why the Wires Matter

All of that splitting comes with a cost. The GPUs have to talk to each other constantly. Tensor parallelism swaps results inside every single layer, and data parallelism averages gradients across the whole cluster on every training step. Any time a GPU is waiting for data from another GPU, it isn't doing math. So the connections between GPUs matter almost as much as the GPUs do. Here is how Meta's cluster was wired:

![Inside one server, eight GPUs connect through NVLink at 900 gigabytes per second per H100, counting both directions. Between servers, each GPU gets a 400 gigabit per second network link. For comparison, a PCIe 5.0 x16 slot in a home PC moves about 128 gigabytes per second counting both directions.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_interconnect.png)

Each of Meta's servers holds eight GPUs connected by NVIDIA's NVLink, which moves up to 900 GB per second per H100, counting both directions. Between servers, every GPU gets its own 400 gigabit per second network link (note the lowercase b, which means bits, so that's 50 GB per second). Meta arranges the work so the chattiest kind of splitting, tensor parallelism, stays inside a server on the fast links, and the less chatty kinds stretch across the network. The company even wrote its own version of NVIDIA's communication library for this run.

A gaming PC has none of that. Its graphics card talks to the rest of the computer over a PCIe slot, which tops out around 128 GB per second counting both directions on the newest PCIe 5.0 slots, and the RTX 4090 doesn't have NVLink at all. Newer data center hardware puts even more GPUs on the fast links. NVIDIA's [GB200 NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/), for example, connects 72 GPUs in a single rack into one NVLink domain.

## Running Nonstop for Weeks

Why does a training run keep going around the clock? Training is sequential in the same way that writing text is. For a given recipe, every update to the weights depends on the update before it, so step 1,000 can't start until step 999 is done. The GPUs can stop and pick up again from a saved snapshot, but every hour they sit idle is an hour of an enormously expensive cluster doing nothing.

What makes this hard is that with tens of thousands of parts, something is always breaking. Meta's paper reports that during one 54-day stretch of training, the job was interrupted 466 times. Of those, 47 were planned, and 419 were unexpected, with about 78% of the unexpected ones caused by confirmed or suspected hardware problems. GPU problems were the single biggest category, at 58.7% of the unexpected interruptions. To survive this, a training job regularly saves a **checkpoint**, a snapshot of the weights and optimizer state, and restarts from the latest one after a failure:

![A day of training with a checkpoint saved every two hours. When something fails, the job restarts from the last checkpoint, and the work done since then has to be redone. Meta reported 466 interruptions in 54 days of Llama 3 training, about one every 3 hours.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_interruptions.png)

The timings in that figure are made up, but the trade-off is real: saving more often means less work to redo, but every save takes time too. Meta reports that its training was doing useful work more than 90% of the time. Checkpoints for a model this size are big, too, and Meta's paper describes a 240-petabyte storage system built to keep up with saving them.

## Power and Cooling

Each H100 in Meta's cluster is rated for up to 700 watts. If we multiply that by 16,384 GPUs, we get about 11.5 megawatts for the GPUs alone, running flat out. That's our own arithmetic and a ceiling, since real power draw goes up and down. On top of the GPUs, there are the CPUs, the network, the storage, and the cooling.

Data centers describe that extra overhead with a number called **power usage effectiveness** (PUE), which is the facility's total power divided by the power that reaches the computing equipment. A PUE of 1.0 would mean zero overhead. The [Uptime Institute's 2024 survey](https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2024.GlobalDataCenterSurvey.Report.pdf) put the industry average at 1.56, meaning about 0.56 watts of overhead for every watt of computing.

Nearly all of that electricity ends up as heat, and it's packed into a small space, so cooling is a big part of the engineering. The densest racks use liquid cooling. The GB200 NVL72 we just mentioned is a liquid-cooled design. Training also creates an odd problem of its own. When thousands of GPUs pause together, say to wait on a checkpoint, the cluster's power draw drops off a cliff and then jumps back. Meta's paper describes what that does to the grid:

> When this happens, it can result in instant fluctuations of power consumption across the data center on the order of tens of megawatts, stretching the limits of the power grid.
>
> [Meta, The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)

## Training vs. Running a Model at Home

None of this means our gaming card is useless for AI. It's just pointed at the wrong job. Running a model that's already trained, which is called **inference**, is a much smaller task:

- Inference only does the forward pass, about 2 operations per parameter for each token, with no backward pass.
- It only needs the weights in memory, with no gradients and no optimizer state.
- It handles one conversation's worth of tokens instead of trillions.

Here are the two jobs side by side:

![Training a model compared with running it, which is called inference. Training takes about 6 operations per parameter per token, keeps the weights, gradients, and optimizer state in memory at about 16 bytes per parameter, and works through trillions of tokens. Inference takes about 2 operations per parameter per token with no backward pass, needs only the weights, and handles one conversation's worth of tokens.](https://www.kirupa.com/ai_fundamentals/images/data_centers_and_training_train_vs_run.png)

There's also a trick for shrinking the weights. **Quantization** stores each weight with fewer bits, say 4 instead of 16, by rounding it onto a coarser scale. Real methods round small groups of weights together, each group with its own scale, so one unusually large weight doesn't ruin its neighbors. Every weight ends up slightly off, and how much quality that costs depends on the model, the method, and the task, so it's something to test. (The [Quantization](https://www.kirupa.com/ai_fundamentals/quantization.htm) article walks through exactly how that rounding works and where it goes wrong.) A popular 4-bit version of Llama 3.1 8B, shared on Hugging Face as a [community quantization](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF), is a 4.92 GB file. Running it takes some extra memory on top of that, including working memory that grows with the length of the conversation, but it fits on a 24 GB card with room to spare.

Even fine-tuning a small model can work at home, especially with techniques like LoRA that train only a small set of extra parameters, which we covered in [Prompting vs. Fine-Tuning](https://www.kirupa.com/ai_fundamentals/prompting_vs_fine_tuning.htm). Training a frontier model from scratch is a different story. That still takes a data center, and the requirements keep climbing. At the time of writing, [Epoch AI estimates](https://epoch.ai/publications/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year) that the training compute of frontier models has been growing by 4 to 5 times per year.

## Conclusion

We started with a simple question: why can't the graphics card in a gaming PC train a model like Llama 3.1 405B? The math answers it pretty decisively. Training takes about 6 operations per parameter per token, which works out to 3.8 × 10^25 operations for this model, or around 7,300 years for one RTX 4090 running perfectly. Before time even becomes the issue, training state of about 16 bytes per parameter needs about 270 times an RTX 4090's memory, and 81 times an H100's.

The way out is to split the work across thousands of GPUs, by data, by layer, and by pieces of each layer. That only works if the GPUs can trade results over very fast links, run nonstop because every step depends on the last, and recover gracefully from the failures that are guaranteed at this scale. All of that draws megawatts of power and turns it into heat that has to go somewhere. Running a trained model is a much lighter job, which is why a quantized model can run happily at home, while training its big siblings stays in buildings full of GPUs.

Some of the newest models spend extra computing power in a second place too: while they answer us, working through a problem before they reply. What that means, and whether it counts as thinking, is where we're going next in [What Does It Mean for AI to Reason?](https://www.kirupa.com/ai_fundamentals/what_ai_reasoning_means.htm)