A customer types this into the chat box on the Crumb & Co. bakery's website: What makes your sourdough sour? Our help desk bot, Biscuit, answers right away:
It talks about the starter, the wild yeast and bacteria living in it, and the long, slow rise that gives the bread its tang. That's a good answer, and a little strange, because we never told Biscuit any of that. The bakery facts we send with every request list our loaves, prices, and hours. They never explain what sourdough is.
So where did that knowledge come from? It came from the large language model (LLM) on the other end of our API call. In the following sections, we'll open that model up and see what's inside, why it's so ridiculously big, what it does and doesn't know, and how its knowledge got there, a story that runs through web crawls, scanned books, a $1.5 billion settlement, human raters, and maybe your own chats.
Onwards!
It All Starts With Guessing the Next Token
Before we get to what an LLM knows, let's look at what it does. We hand it some text, chopped into tokens (small chunks of text, often a whole word or a piece of one, which the next article looks at closely), and it guesses which token comes next. The guess isn't a single answer. It's a score for every token in the model's vocabulary (the full list of tokens it knows), and those scores work out to odds. Here is what that might look like for one short bit of text:
The odds are made up, but the shape is real. To write a reply, the model picks one token from those odds (how adventurous that pick is comes down to a setting called temperature, which gets its own article later in this series), adds it to the text, and runs everything again to get odds for the token after that. OpenAI describes its own models the same way: they predict "the next most likely word when generating a response, one word at a time".
That sounds too simple to explain Biscuit's sourdough answer, until we think about what it takes to guess well. To give starter good odds after Sourdough is made with a, the model has to have picked up something about sourdough, and guessing well across every kind of text requires knowing a lot about the world. So basically, the knowledge Biscuit showed off is a side effect of the model getting very VERY good at guessing.
A Giant Pile of Numbers
Where does a model keep all that knowledge? A natural guess is a big table of facts, like a database with a row for sourdough. Nope. If we opened up the model's files, here is what we would find instead:
An LLM is a giant pile of numbers called parameters (aka weights). Our tokens go in as lists of numbers, get multiplied and added against those weights layer after layer, and come out the other end as a score for every possible next token. The model never looks up sourdough in an index or searches the web. (Chat apps that search the web do it with a separate tool, which Tool Calling covers.) What the model knows about sourdough is smeared across billions of numbers, mixed in with everything else it knows, and no single number means anything on its own.
How many numbers are we talking about? Meta's Llama 3.1, one of the few large models whose details are public, comes in a version with 405 billion parameters. At 2 bytes per number, a common format, that's about 810 GB of weights. That's why the biggest models run on racks of specialized hardware, and why tricks like quantization, which stores each number in fewer bytes, exist.
Why They're So Big
Every weight starts out as a random number. Training shows the model real text with the next token hidden, checks its guess against what actually came next, and nudges every weight a tiny bit in whichever direction would have made the right token more likely. After that happens across trillions of tokens, the weights settle into values that guess well.
This is where size comes in. More weights give the model more room to store patterns, and more text gives it more patterns to learn. In 2020, researchers at OpenAI found that a model's error drops smoothly and predictably as the model, the data, and the computing power grow. Two years later, DeepMind's Chinchilla paper showed that the training data has to grow along with the model. Here is how that played out between GPT-3 and the largest Llama 3.1:
In four years, the parameter count a little more than doubled, but the training text grew about 50 times. Meta trained that model on up to 16,000 H100 GPUs (chips built for exactly this kind of math), for about 3.8 × 1025 floating-point operations, which are individual multiplications and additions, in total. Bills like that are why only a handful of companies train models this big, and size is also why big models cost more per token (Small vs. Large Model Trade-offs digs into that). We're using these two because many newer models, like GPT-4, don't publish these numbers at all.
What's Actually in There
Let's do some very rough math. At OpenAI's rule of thumb of about four characters per token, Llama 3.1's 15.6 trillion training tokens are something like 60 terabytes of text. Its weights are about 810 gigabytes, around 75 times smaller than what it read. This means the model can't be storing its training text word for word. What it keeps is closer to the patterns in that text, including the facts that come up again and again. Here is a quick sort of what we can and can't expect to find in the weights:
Sourdough shows up in countless recipes, blogs, and baking forums, so that pattern is strong and Biscuit explains it confidently. Crumb & Co.'s menu never appeared in any training data, which is exactly why we send the bakery facts with every request, as we saw in The Anatomy of a Chat Request. A few famous behaviors fall out of this too:
- When a fact is rare or fuzzy in the weights, the model still produces the most plausible-sounding tokens, and plausible isn't the same as true. That's a hallucination. OpenAI's researchers argue that the way models are trained and tested rewards a confident guess over an honest I don't know.
- The training data stops at some date, called the knowledge cutoff. For Llama 3.1, it's December 2023, and anything after that isn't in the weights.
- Text that appears many, many times can get memorized almost word for word. Researchers pulled hundreds of verbatim snippets out of GPT-2, and The New York Times's lawsuit against OpenAI includes an exhibit titled "One Hundred Examples of GPT-4 Memorizing Content from The New York Times".
All of it, patterns and memorized bits alike, comes from the training text. So where did that text come from?
How the Knowledge Gets In
A chat model like the one behind Biscuit is built in two big stages. Here is the journey from raw text to the model we talk to:
Pretraining plays the guessing game across an enormous pile of text, and it's where nearly all of the knowledge comes from. Post-training then teaches the result to act like an assistant.
The Open Web
Most pretraining text comes from the public web. The best-known source is Common Crawl, a nonprofit founded in 2007 that gives away its archive of the web for free. At the time of writing, it holds over 300 billion pages and adds 3 to 5 billion more each month. Filtered Common Crawl made up 60% of GPT-3's training mix, and a 2024 Mozilla Foundation study found that at least 64% of 47 LLMs it reviewed used it too. Companies filter out duplicates, spam, and junk before training.
Code and Wikipedia
Code is a big slice too, about 17% of Llama 3's training tokens, and OpenAI's 2021 Codex model learned from 54 million public GitHub repositories. Wikipedia, with more than 7.2 million English articles at the time of writing, is tiny next to the web but carefully edited. In GPT-3's mix, it was only 3% of the total, yet the model went through it about 3.4 times while seeing less than half of its Common Crawl data. The people training a model decide how much each source counts, and the cleaner ones get counted more.
Books, Bought and Pirated
Books are especially valuable because they're long, edited, and full of sustained writing. They're also copyrighted, and how AI companies got hold of them has ended up in court.
Early on, a lot of book text came from pirated collections. The Pile, a popular open dataset from 2020, included a set called Books3 with 196,640 books copied from Bibliotik, a private tracker for pirated books. After The Atlantic published a searchable list of its authors in 2023, a Danish anti-piracy group got its best-known copy taken down. Court records in a lawsuit authors brought against Meta show the company downloaded the pirate libraries LibGen and Anna's Archive after its attempts to license books stalled.
Anthropic's case went the furthest. In Bartz v. Anthropic, Judge William Alsup found in June 2025 that Anthropic had downloaded more than seven million pirated copies of books. The same ruling describes a second effort. In February 2024, Anthropic hired the former head of partnerships for Google's book-scanning project, who was tasked with obtaining "all the books in the world". The company spent many millions of dollars on millions of print books, often used ones, and its service providers stripped off the bindings, cut the pages, scanned them, and threw away the paper originals.
The judge drew a sharp line between the two. Training Claude on the books was fair use, which he called "exceedingly transformative", and so was scanning the print books Anthropic bought. Keeping millions of pirated copies in a permanent library was not, and that part was headed for trial. Anthropic settled instead, agreeing to pay $1.5 billion, about $3,000 for each of roughly 482,000 books, and to destroy the pirated files. A new judge gave the deal final approval in July 2026, and Reuters called it the largest known settlement of a U.S. copyright case. Meta also won on fair use in its case, but Judge Vince Chhabria was careful to say what his ruling meant:
This ruling does not stand for the proposition that Meta's use of copyrighted materials to train its language models is lawful. It stands only for the proposition that these plaintiffs made the wrong arguments and failed to develop a record in support of the right one.
Licensed Data
More and more data now comes through deals. OpenAI licensed part of the Associated Press's news archive in 2023, and Reddit agreed to let Google train on its content in a deal reportedly worth about $60 million a year. The terms of most of these deals aren't public, so the dollar amounts usually come from news reports.
Post-Training and the People Behind It
A model fresh out of pretraining (a base model) is a brilliant autocomplete, but it isn't an assistant. If we ask it a question, it might just continue with more questions, the way a quiz page would. Turning it into something like Biscuit takes post-training, and post-training runs on people.
OpenAI's 2022 InstructGPT paper laid out the classic recipe. Human labelers wrote example answers for the model to imitate. Then they ranked the model's own answers from best to worst, those rankings trained a reward model that predicts which answer people prefer, and the chat model was tuned to score well with it. This is reinforcement learning from human feedback (RLHF). OpenAI used about 40 contractors, and people preferred answers from the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3, a model more than 100 times larger.
Here is that recipe, step by step:
Today's scale takes far more people, and a whole industry supplies them. The best-known company is Scale AI, which has prepared training data for OpenAI, Google, and Microsoft using contractors around the world. In 2023, The Washington Post reported that many of its Remotasks workers in the Philippines complained of low pay and said they weren't always paid. In June 2025, Meta put $14.3 billion into Scale and hired its founder, Alexandr Wang, to lead Meta's AI efforts. OpenAI had already been winding down its work with Scale, Google was reportedly cutting ties, and a month later Scale laid off 14% of its staff.
Where Our Chats Come In
There's one more source of training data, and it might be you. Conversations in consumer AI apps can become training data for future models. Apps built on an API, like Biscuit, are a different story. Here is how the two paths compare:
The details vary by company and change often. Here is where things stand at the time of writing:
- OpenAI may train on conversations from ChatGPT's Free, Plus, and Pro plans unless we turn off Improve the model for everyone under Settings > Data controls. Temporary chats aren't used for training. One catch is that a thumbs up or thumbs down on a reply lets OpenAI use that whole conversation even after we've opted out.
- Since August 2025, Claude Free, Pro, and Max users choose whether Anthropic trains on their chats and coding sessions. If they allow it, Anthropic keeps that data for five years instead of 30 days, and thumbs-up or thumbs-down feedback may be used for training too.
- Google uses Gemini app chats to improve its AI models while the Keep Activity setting is on, and human reviewers may read some chats.
APIs and business plans are treated differently. OpenAI doesn't train on its API or ChatGPT Business, Enterprise, and Edu by default, and Anthropic's consumer change doesn't cover its API or Claude for Work, Government, or Education. That's good news for Biscuit, because our customers' messages go through the API. It's still worth reading our provider's terms and not sending personal details the model doesn't need (PII Redaction shows how).
Notice the word future in the figure. A chat used for training doesn't teach today's model anything, because its weights are frozen. It only shapes a model built later, and Anthropic notes that models released today began development 18 to 24 months earlier.
Keeping Your Chats Out of Training
If you'd rather your conversations stay out of future models, a few habits go a long way:
- Check the training setting in every AI app you use, and again after any policy update.
- Use a temporary chat for anything sensitive. Claude calls it an incognito chat.
- Skip the thumbs-up and thumbs-down buttons on conversations you want kept out.
- Don't paste passwords, keys, or other people's personal information into any chat.
- If you run a website, block AI crawlers like OpenAI's GPTBot and Anthropic's ClaudeBot in robots.txt.
A setting only changes what happens from then on. Anthropic, for example, says chats it has already used stay in models that are already trained or in training.
Conclusion
When Biscuit explained sourdough, it wasn't looking anything up. It was guessing one token at a time with billions of numbers, tuned on trillions of tokens of web pages, code, Wikipedia, and books, then shaped by people rating answers. Those numbers hold patterns more than facts, which is why the model knows sourdough, might make up a rare detail, and has never heard of Crumb & Co.
The other detail to keep in mind, and this is a HUGE detail, is that the weights are frozen once training ends, and the model doesn't remember anything between requests either. Everything it knows about this customer, this bakery, and this conversation has to arrive in the request itself, and all of it reaches the model as tokens. We've been using that word loosely, so in Tokens and How They Set Your Bill, we'll see exactly what a token is and why tokens decide what every request costs.
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
