Quantization

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

A language model is, underneath everything, a giant pile of numbers. A 7-billion-parameter model has about seven billion of them, and every single one has to sit in memory while the model runs.

That's a problem when the memory is small. Quantization is the trick of storing each of those numbers with fewer bits, so the whole pile takes less space. It sounds a little like cheating, and in a way it is, because we're deliberately making every number slightly wrong. Even so, the model often behaves almost exactly the same. (We got a quick preview of this in Why Training Takes a Data Center (https://www.kirupa.com/ai_fundamentals/data_centers_and_training.htm), where it let a model run on a gaming card at home.)

In this article, we'll squeeze a real-sized model onto a laptop, quantize four numbers by hand to see exactly what's lost, and run into the one weight that ruins everything.

Onwards!

Nadia's Laptop

Let's say Nadia wants to run an open-weights model (one whose weights anyone can download and run themselves) on her laptop so she can work offline. The model has 7 billion parameters. Her laptop's GPU has 8 GB of memory.

Models are commonly distributed with each parameter stored as a 16-bit number, which takes 2 bytes. The arithmetic is short:

- 7 billion parameters × 2 bytes = 14 GB

That doesn't fit in 8 GB. Here's how it looks at a few different sizes per number:

[Image: A 7-billion-parameter model needs 14 GB at 16 bits, too big for Nadia's 8 GB GPU, 7 GB at 8 bits, which is tight, and 3.5 GB at 4 bits, which fits comfortably.] (https://www.kirupa.com/ai_fundamentals/images/quantization_memory_fit.png)

At 8 bits per number, the weights take 7 GB, which technically fits but leaves almost no room for anything else (and the model needs working memory while it runs). At 4 bits, the weights take 3.5 GB, with comfortable room to spare. (Real 4-bit files come out a little bigger than that, for a reason we'll see shortly.)

So the question becomes: how do we store a number like -0.53 in 4 bits, and what do we lose?

Rounding to the Nearest Mark

Imagine a ruler. A good ruler has lots of tiny marks, so we can measure precisely. A cheap ruler has only a few marks, so every measurement gets rounded to the nearest one. Quantization is choosing a ruler with fewer marks for the model's numbers.

Let's do it by hand with four weights from Nadia's model:

0.12, -0.53, 0.97, -0.08

To build the ruler, we look at the largest weight by size, which is 0.97. We stretch the ruler so its last mark lands exactly there. With 8 bits, we'll use whole numbers from -127 to 127, so each step on the ruler is worth 0.97 / 127, about 0.0076. That step size is called the scale.

Then each weight is divided by the scale and rounded to the nearest whole number. That whole number is what we store. When the model runs, it multiplies back by the scale to get an approximate weight.

Here is how our four weights land on the 8-bit ruler and on a 4-bit one, where we only get whole numbers from -7 to 7:

[Image: The weights 0.12, -0.53, 0.97, and -0.08 snapping to the nearest mark on two rulers. On the 8-bit ruler with marks from -127 to 127, each lands almost where it started. On the 4-bit ruler with marks from -7 to 7, they land on 1, -4, 7, and -1, and the small weights move noticeably.] (https://www.kirupa.com/ai_fundamentals/images/quantization_rulers.png)

And here are the actual numbers:

Original | 8-bit stored | 8-bit restored | 4-bit stored | 4-bit restored
0.12 | 16 | 0.1222 | 1 | 0.1386
0.97 | 127 | 0.9700 | 7 | 0.9700

At 8 bits, every weight comes back within a few thousandths of where it started. At 4 bits, the errors are much bigger. The -0.08 weight comes back as -0.1386, which is nearly double its original size.

What's noteworthy is that the model has billions of these tiny errors, and they mostly wash out. Each weight is one small contribution to sums over thousands of other weights, and many of the rounding errors partly cancel each other. That's why 8-bit versions of models often behave almost the same as the 16-bit originals, and why 4-bit versions are often still very usable. I said often on purpose, and we'll come back to that.

The Weight That Ruins Everything

Our toy example had a polite set of weights. Real models have a less polite habit: every so often, there's a weight (or an activation, one of the in-between numbers the model works out as it runs) that's much larger than its neighbors.

Let's say our third weight was 9.0 instead of 0.97:

0.12, -0.53, 9.0, -0.08

Now the ruler has to stretch all the way to 9.0. At 4 bits, each step is worth 9.0 / 7, about 1.29. Every other weight is far smaller than half a step, so they all round to zero:

[Image: A 4-bit ruler stretched to fit a weight of 9.0. The 9.0 lands on mark 7, while 0.12, -0.53, and -0.08 all snap to the 0 mark. A handwritten note says all zero!] (https://www.kirupa.com/ai_fundamentals/images/quantization_outlier.png)

One outlier just erased three weights. If that happens across a whole model, the damage stops washing out.

Our example only had four weights, so one ruler covered all of them. A real model has millions of weights in each layer. If a whole layer shared one ruler, a single outlier would flatten a huge number of its neighbors.

The fix is to stop using one ruler for everything. Instead, we split the weights into small, fixed-size groups (often a few dozen to a hundred or so weights each), and each group gets its own scale. The outlier still flattens the small weights that share its group. What changes is that the damage stops at the edge of that group, and every other group keeps a ruler sized to its own weights. Storing all those extra scales is the reason real 4-bit model files come out a bit bigger than our 3.5 GB estimate.

Here's the idea with two groups of four, where our original four weights are Group A and the outlier lands in Group B:

[Image: Two groups, each with its own ruler. Group A, with a scale of about 0.14, keeps 0.12, -0.53, 0.97, and -0.08 on four different marks. Group B, with a scale of about 1.29, holds the 9.0 outlier, and its small weights 0.05, -0.21, and 0.44 all land on 0.] (https://www.kirupa.com/ai_fundamentals/images/quantization_groups.png)

Real methods also deal with outliers more directly. LLM.int8() (https://arxiv.org/abs/2208.07339), for example, does the math for the rare outlier features in 16 bits and runs everything else in 8 bits.

There are also smarter ways to decide the rounding itself. GPTQ (https://arxiv.org/abs/2210.17323) quantizes weights a few at a time and adjusts the not-yet-quantized weights to make up for the rounding error so far, using a small set of sample data to judge what matters. AWQ (https://arxiv.org/abs/2306.00978) runs sample data through the model, finds the small share of weights that the activations lean on most, and scales them so they keep more precision after rounding.

If Nadia downloads a quantized model for a laptop tool built on llama.cpp, it'll probably come as a GGUF (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) file. GGUF is a file format rather than a quantization method, and it carries llama.cpp's own quantization types, with names like Q4_K_M or Q8_0 that hint at roughly how many bits each weight gets.

Why It's Faster, Too

Smaller isn't the only win. When a model generates text, it produces one token at a time, and for each token the GPU has to stream the weights from memory. That memory trip, more than the math, is usually what limits how fast tokens come out. (The Request Batching (https://www.kirupa.com/ai_fundamentals/request_batching.htm) article walks through why.)

A 4-bit model has a quarter of the bytes to move compared with a 16-bit one. Less to move generally means tokens come out faster. The exact speedup depends on the hardware and on whether the software has efficient routines for the quantized format, so it's something to measure rather than assume.

Here is the difference in what moves for each token:

[Image: Two bars for the 7-billion-parameter model. At 16 bits, the GPU streams about 14 GB of weights for every token. At 4 bits, it streams about 3.5 GB, a quarter as much. The real speedup depends on the hardware and software, so measure it.] (https://www.kirupa.com/ai_fundamentals/images/quantization_bytes_per_token.png)

Checking What We Lost

Here's the honest part. Usually barely notices is a statement about averages across many models and many tasks. It isn't a promise about your model on your task. The amount of quality lost depends on the model, the bit width, the quantization method, and the kind of work you ask it to do.

So the right move is the same one we'd use for any model change: run the quantized model through the same evaluation set as the original and compare. (A golden dataset (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) is exactly that kind of set.) If Nadia uses her model to summarize meeting notes, she should test it on meeting notes, not on a general benchmark. If the 4-bit version holds up, great. If it doesn't, an 8-bit version or a smaller model at higher precision might be the better trade.

Conclusion

Quantization stores each of a model's numbers with fewer bits by rounding them onto a coarser ruler and keeping a scale to stretch them back. Every number gets a little wrong, and across billions of weights those errors mostly wash out. The trouble comes from outliers, which stretch the ruler until smaller weights round away. That's why real methods use small groups with their own scales, treat outliers specially, and use sample data to decide how to round.

For Nadia, that meant a 14 GB model that wouldn't fit became a roughly 4 GB model that runs comfortably on her laptop, and generates tokens faster because there's less to move through memory for each one. The one thing she can't skip is testing it on her own work, because the only quality loss that matters is the one on the task she actually cares about!