Language models are great at reading messy human writing. Our code is not. Code wants data in a precise shape, with exact key names, exact types, and only certain allowed values. Getting from one world to the other is one of the most common jobs we give a model, and it's also where a lot of apps break without anyone noticing right away.
In the following sections, we'll look at why please reply in JSON isn't enough, how a JSON schema lets the model's output be guaranteed to have the shape we want, how that guarantee actually works under the hood, and the one big thing it still doesn't promise.
Onwards!
The Bakery Inbox
Crumb & Co., the bakery behind our help desk bot Biscuit, also takes custom orders by email. Every morning, somebody reads the inbox and types each order into the kitchen's order system. We'd like a model to do that typing for us. Here's a typical email, sent on Wednesday, November 11, 2026:
Hi! Could I get two dozen cinnamon rolls and one sourdough loaf for pickup this Friday morning? Name's Tomás. Thanks!!
What the kitchen system needs is a record that looks like this:
{
"customer": "Tomás",
"items": [
{ "product": "cinnamon_roll", "quantity": 24 },
{ "product": "sourdough_loaf", "quantity": 1 }
],
"pickup_date": "2026-11-13"
}There are some small decisions hiding in there. Two dozen became the number 24. This Friday became an actual date. And cinnamon rolls became cinnamon_roll, which is the exact product code the kitchen system expects.
Asking Nicely
The first thing everyone tries is to describe the format in the prompt: Reply with JSON containing customer, items, and pickup_date. If we run that a hundred times, most of the outputs will be fine. Some won't be, and they tend to go wrong in a few familiar ways:
Each of these breaks our code differently. The first two fail to parse at all. The third one parses just fine and then blows up later, when the kitchen system tries to multiply two dozen by the price of a cinnamon roll. That third kind is the sneakiest, because nothing complains right away.
We could write cleanup code that strips code fences and hunts for the first curly brace. People do! It gets ugly fast, though, and no amount of cleanup can fix a key with the wrong name.
Describing the Shape With a Schema
A better approach is to describe the exact shape we want in a format that a program can check. That format is JSON Schema, a standard way of writing down which keys an object has, what type each value is, and which values are allowed. Here's a schema for a Crumb & Co. order:
{
"type": "object",
"properties": {
"customer": { "type": "string" },
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"product": {
"type": "string",
"enum": ["cinnamon_roll", "sourdough_loaf", "baguette", "croissant", "other"]
},
"quantity": { "type": "integer" }
},
"required": ["product", "quantity"],
"additionalProperties": false
}
},
"pickup_date": { "type": ["string", "null"] }
},
"required": ["customer", "items", "pickup_date"],
"additionalProperties": false
}The enum says product must be one of five exact strings, so the model can't write Cinnamon Rolls or cinn roll. The integer type means quantity can't be two dozen. And additionalProperties: false means no surprise keys show up. (We'll come back to why pickup_date is allowed to be null and why there's an "other" product in a bit.)
Most providers let us send a schema like this along with the request. With OpenAI's API, for example, we pass it as a JSON schema response format with strict: true, and Anthropic and Google have their own equivalents. In Python and TypeScript, we usually don't write the schema by hand. We describe the order with a Pydantic or Zod model, and the SDK turns it into a schema for us.
How the Guarantee Works
So how can a provider guarantee that the output matches? The model is still just predicting one token at a time. Couldn't it still wander off?
Not when the schema is enforced. At every step, a model scores every token in its vocabulary and then picks one (Temperature for Predictable Output walks through that pick). With structured outputs, the provider first turns our schema into a set of grammar rules. Then, after every token, it checks which tokens the rules allow next, and it sets the probability of every other token to zero before the pick happens. This is called constrained decoding. Here's what it looks like at one moment while the model is writing Tomás's order:
Notice what happened there. The model's favorite next token was the start of the word two. Left alone, it would have written two dozen as a string. The grammar says an integer has to come next, so the quote mark is off the table, and the best token that's still allowed is 24. Since a blocked token can never be picked, the model has no way to write output that breaks the schema. It can't put a string where the schema wants an integer, and it can't close the object before every required key is there.
OpenAI described exactly this approach when it added the feature to its API in August 2024. In that announcement, OpenAI reported that on its own test of following complex schemas, its newest model with structured outputs scored 100%, while an older GPT-4 model that was only asked nicely scored under 40%. It also mentions one cost: the very first request with a new schema takes longer, because the grammar has to be built before the model can use it.
Note: JSON mode is a weaker cousin
Some APIs also offer a JSON mode. It promises that the output is valid JSON, so it rules out the code fence and the chatty preamble. It says nothing about which keys appear or what types they hold, though, so our third broken example could still happen. When a schema option is available, prefer it.
Right Shape, Wrong Answer
There's a catch, though, and it's a big one. A schema guarantees the shape of the answer. It says nothing about whether the answer is true. Look at what happens with this email:
Two baguettes for Ana please, I'll swing by sometime.
Ana never says when. If our schema had made pickup_date a required string, the model would have to write some string, and a model that has to write a date will happily write a plausible one, like tomorrow's. The output passes every check, even though the date is made up. Compare what we get from the two versions of the schema:
That's why our schema lets pickup_date be null. Giving the model an honest way to say not provided is one of the most important parts of designing a schema. The "other" product does the same job. If a customer orders a lemon tart, which isn't on our list, it's much better for the model to say "other" than to be forced into picking the closest wrong product.
Even with a careful schema, we still check the values in our own code. Is every quantity greater than zero? Is the pickup date in the future? Does Tomás's this Friday actually match the date the model wrote? The schema takes care of the format, and our code checks that the content makes sense.
A few other details are worth knowing before we lean on this:
- Each provider supports a subset of JSON Schema, and the subsets differ. A keyword like minimum or pattern may be ignored or rejected, depending on the provider, so check the docs before relying on one.
- The model can still refuse. If a request trips a safety check, some APIs send back a refusal instead of the object, and our code needs to handle that case.
- If the reply gets cut off by the output token limit, we can end up with an unfinished object. Set the limit high enough for the biggest order we expect.
Conclusion
Asking a model for JSON in plain words works most of the time, and most of the time is a rough place for a data pipeline to live. A JSON schema describes the exact shape our code needs, and when the provider enforces it with constrained decoding, tokens that would break the shape are never picked. This means our code no longer needs cleanup logic for code fences, chatty preambles, or wrong key names.
What a schema can't do is make the model honest. Designing the schema with ways out like null and "other", and then checking the values in our own code, keeps the model from filling required fields with confident guesses.
The same idea of handing the model a schema shows up again when we want the model to ask our code to do something, like checking whether there are any cinnamon rolls left before promising some to a customer. That's called tool calling, and it's where we'll go next.
Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.
Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.
Your support keeps this site going! 😇
