If we ask a language model the same question twice, we might get two different answers. Sometimes that's exactly what we want, like when we're brainstorming names for a new cake. Other times it's maddening, like when we're pulling the total out of an invoice and the answer should be the same every single time.
The setting that controls a lot of this variety is called temperature. It's a single number we can send with every request. On OpenAI's API, for example, it runs from 0 to 2 and defaults to 1. To understand what it really does, we need to peek at the very last step a model performs before it writes each token.
Onwards!
The Model's Prize Wheel
A language model writes one token at a time. For each next token, the model doesn't come up with a single answer. It produces a score for every token in its vocabulary, and those scores get turned into probabilities that add up to 100%.
Let's say the model is completing the sentence The best pizza topping is. To keep things readable, we'll pretend only four tokens are in the running, with these made-up probabilities:
| Next token | Probability |
|---|---|
| mushrooms | 46% |
| pepperoni | 31% |
| pineapple | 15% |
| anchovies | 8% |
Now the model has to pick one. It doesn't automatically take the top choice. It samples, which is a fancy way of saying it spins a prize wheel where each token gets a slice sized by its probability:
Most spins land on mushrooms. But almost a third of the time we get pepperoni, and now and then we get anchovies. Since the model spins again for every token after that, one different pick early on can send the rest of the answer somewhere completely different. That's where the variety comes from.
What Temperature Changes
Temperature reshapes the slices before the wheel spins. It doesn't change which token the model likes best. It changes how much more likely the favorite is compared to the rest.
Under the hood, the model's raw scores (aka logits) get divided by the temperature before they're turned into probabilities. Dividing by a small number stretches the gaps between the scores, so the leader pulls even further ahead. Dividing by a large number squishes the gaps, so everything moves closer to even. A temperature of exactly 1 leaves the scores alone. Here is what that does to our four toppings, starting from the same raw scores each time:
| Temperature | mushrooms | pepperoni | pineapple | anchovies |
|---|---|---|---|---|
| 0.2 | 88% | 12% | under 1% | under 1% |
| 0.5 | 63% | 28% | 7% | 2% |
| 1.0 | 46% | 31% | 15% | 8% |
| 1.5 | 39% | 30% | 19% | 12% |
And here is that same table drawn as wheels:
At 0.2, the wheel is basically rigged, and we'll see mushrooms almost every time. At 1.5, pineapple and anchovies show up a lot more often. Neither setting makes the model smarter. A low temperature means the model plays its favorite more often, and a high temperature means it takes more chances.
If you'd like to check the math, the four raw scores behind every row of the table are 2.0, 1.6, 0.9, and 0.2. Divide each one by the temperature, raise e to that power, and then divide each result by their total. That last recipe is called softmax, and it's how almost every model turns scores into probabilities.
Temperature 0
What if we turn the temperature all the way down to 0? Dividing by zero doesn't work, so APIs treat 0 as a special case. They skip the wheel and always pick the top token, which is called greedy decoding. In our example, the model says mushrooms every time.
So temperature 0 means perfectly identical answers, right? As it turns out, not quite.
In a September 2025 post called Defeating Nondeterminism in LLM Inference, Horace He and Thinking Machines Lab sent the same prompt to an open model 1,000 times at temperature 0 and got back 80 different completions. All of them were identical for the first 102 tokens before they started to drift apart. The cause turned out to be on the server side. Depending on how many other requests are being processed at the same moment, the server adds up the model's numbers in a slightly different order, and computers round decimal numbers in ways that depend on that order. When two tokens are nearly tied, a tiny rounding difference is enough to flip which one wins. Here is that near tie in picture form:
This means temperature 0 gets us much more consistent answers, but we shouldn't build anything that breaks when two runs differ by a word.
Note: Some models don't take a temperature at all
Reasoning models often manage their own sampling. At the time of writing, OpenAI's guide for its latest models says to remove temperature and top_p from the request whenever reasoning is turned on. If a request fails with an unsupported parameter error after you switch models, this is probably why. Check your provider's documentation for the specific model you're using.
A Close Cousin: Top-P
Right next to temperature, you'll often see a setting called top-p (aka nucleus sampling). Instead of reshaping the slices, top-p removes the smallest ones. It keeps the most likely tokens until their probabilities add up to p, throws out everything else, and then resizes the remaining slices to fill the wheel.
With our temperature 1.0 wheel and a top-p of 0.9, the model keeps mushrooms (46%), adds pepperoni (77% so far), adds pineapple (92% so far), and stops there because it has passed 90%. Anchovies get cut. This is a handy way to stop truly weird tokens from ever showing up while still allowing some variety. OpenAI's API reference recommends adjusting temperature or top-p, but not both.
Here is top-p working its way down the wheel:
Choosing a Temperature
So how do we pick? It mostly comes down to whether the task has one right answer or many good ones:
For extraction, classification, routing, and code, we want the model's best guess every time, so a low temperature is a good default. For brainstorming or writing alternatives, where we'd happily show someone five different options, a higher temperature gives us that spread. And if we never touch the setting at all, we get the model's default amount of variety, which is often a temperature of 1.
There's one more thing worth keeping in mind. Temperature only controls how the model chooses among the tokens it already thinks are likely. It can't make a confused model correct. If our prompt is vague, temperature 0 just makes the model consistently vague. For predictable output, a clear prompt and a fixed output format do more than any temperature setting, and locking down that format is exactly what Structured Outputs with JSON Schemas is about.
Conclusion
Every token a model writes comes from a weighted spin of a wheel. Temperature reshapes that wheel before the spin. Low values make the favorite win almost every time, and high values give the long shots a real chance. Temperature 0 skips the spin entirely, though rounding differences on the server can still make two runs drift apart. Top-p takes a different approach and trims off the least likely options.
Use a low temperature when there's one right answer and a higher one when variety is the point. And when an answer really has to be predictable, don't lean on temperature alone. Pair it with a clear prompt, a strict output format, and tests that check what comes back (Regression Tests on Every Prompt Change shows how to set those up).
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
