Batch vs. Real-Time Inference

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

Mailing a letter is a lot cheaper than sending a courier. Both get the letter there. The difference is that the courier drops everything for you right now, while the post office gets to bundle your letter with thousands of others and send them along when it's convenient. You pay for the urgency.

Model providers sell both of these for inference, which is just the term for running a model to get an answer. Real-time inference is the courier: we send a request, and the answer comes right back. Batch inference is the post office: we hand over a big pile of requests, and the answers come back later, at a discount. In the following sections, we'll look at one store that needs both, see how a batch job actually works, and figure out how to tell which kind of work we're looking at.

Onwards!

One Store, Two Kinds of Work

Dog-Eared Books is an online bookstore, and it uses models in two very different ways.

The first is an Ask about this book assistant on every product page. A shopper types Is this too scary for a ten-year-old? and waits for an answer. Every second they wait is a second they might wander off.

Here is that assistant, sitting in the corner of one of the store's product pages:

[Image: The Ask about this book assistant sits in a chat panel in the corner of a Dog-Eared Books product page. The panel header shows a robot avatar with a stack of books badge, the name Ask about this book, and the tagline Dog-Eared Books. A shopper asks, Is this too scary for a ten-year-old? The assistant replies, It has a few spooky scenes, but mostly it's a fun adventure! A handwritten note says The book helper!] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_meet_book_assistant.png)

The second is a cleanup project. The store has 40,000 books in its catalog, and many of them have missing or messy descriptions. The team wants a model to write a short, friendly summary for each one. Nobody's sitting there watching. If the summaries show up tomorrow morning, that's great. Here are those two jobs side by side:

[Image: Two lanes. Real-time: a shopper asks a question, the model answers through the real-time API, and the answer arrives in seconds. Batch: 40,000 books that need summaries go to the model through the batch API, and the summaries arrive by tomorrow.] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_two_lanes.png)

The shopper's question is real-time work. The catalog summaries are batch work. The rest of this article is about why that difference matters so much.

Doing the Catalog the Real-Time Way

The obvious first move is to write a loop that calls the normal API 40,000 times. That works, but it bumps into two problems.

The first is rate limits. Our account can only send so many requests and tokens per minute. Our loop has to tiptoe along under that limit, and while it's running, it's using up the same limit the live assistant needs. A shopper's question could end up waiting behind summary number 23,417.

Here is what that looks like for the shopper:

[Image: The Ask about this book chat window on a Dog-Eared Books product page. A shopper asks, Is this too scary for a ten-year-old? The assistant shows three typing dots. A handwritten note says: Behind summary #23,417!] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_chat_waiting.png)

The second is price. Every one of those 40,000 calls pays the full real-time rate, even though none of them needed a real-time answer.

How a Batch Job Works

Several providers offer a separate batch API for exactly this situation. Instead of 40,000 calls, we make one upload. The steps look like this:

[Image: A batch job in four steps. 1, write a file, with a custom_id such as book-00001 on each request. 2, submit it, and the status says validating. 3, wait, while the status says in_progress. 4, download results, which may come back as book-00002 before book-00001. A handwritten note says order not guaranteed.] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_steps.png)

We write all of our requests into one file, one request per line (usually a format called JSONL, where each line is its own JSON object). Each line gets a custom_id that we choose, like book-00001. Then we submit the file and get back a batch ID. Now we wait. When the batch is finished, we download a results file.

That custom_id is more important than it looks. Both OpenAI's and Anthropic's docs warn that results may not come back in the same order as the requests. We match every result to its book using the ID, never by its position in the file.

What the Wait Buys Us

So what do we get for being patient? At the time of writing, OpenAI's Batch API (https://developers.openai.com/api/docs/guides/batch) and Anthropic's Message Batches API (https://platform.claude.com/docs/en/build-with-claude/batch-processing) both charge 50% less than their regular, real-time rates. OpenAI says each batch completes within 24 hours. Anthropic says most batches finish in under an hour, and a batch that isn't done within 24 hours expires. OpenAI also runs batches against their own, higher rate limits, so the catalog job stops competing with the live shopper assistant. (These details change, so check your provider's current docs before planning around them.)

To make the money part concrete, let's use made-up numbers. If summarizing all 40,000 books in real time would cost $200, the same job as a batch would cost about $100. Nothing about the summaries themselves changed. We used the same model and the same prompt, and we got the same kind of output. We just told the provider it could fit our work in whenever it's convenient, and that flexibility is what the discount pays for.

Here are the two bills side by side:

[Image: Two bars. Summarizing all 40,000 books in real time costs 200 dollars. The same job as a batch costs about 100 dollars. Same model, same prompt, same kind of output.] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_cost.png)

Don't Batch 40,000 Mistakes

A batch makes it very easy to make one mistake 40,000 times. If our prompt has a bug, like accidentally including spoilers or getting the reading level wrong, we won't find out until the batch finishes, and by then we've paid for 40,000 bad summaries. So before submitting the big batch, run a small sample in real time:

[Image: Fifty books go through the real-time API, and you review the summaries. Once they look right, the remaining 39,950 books go out as a batch.] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_sample_first.png)

Fifty books in real time costs very little, and we see the results in a minute. Once those look right, we send the rest as a batch.

Results files also come with failures mixed in. A few lines will come back as errors, or as expired if the batch ran out of time. Our code should collect those IDs and resubmit just those, rather than assuming everything worked.

Which Lane Does This Work Belong In?

The question that sorts almost everything is a simple one. Is a person waiting on this answer right now? Here is how that question, plus one follow-up, splits up the work:

[Image: A decision guide. Is someone waiting right now? If yes, use real-time. If no, can it wait for hours? If yes, use a batch API. If no, use your own queue with real-time calls.] (https://www.kirupa.com/ai_fundamentals/images/batch_vs_realtime_inference_decision.png)

If someone's waiting, it's real-time. Chat, autocomplete, the shopper's question, and anything a user just clicked belong here, and long answers should be streamed (https://www.kirupa.com/ai_fundamentals/streaming_responses.htm).

If nobody's waiting and the work can sit for hours, it's a great batch candidate. Catalog descriptions, tagging old support tickets, generating embeddings (https://www.kirupa.com/ai_fundamentals/embeddings.htm) for a big pile of documents, and running evaluation sets all fit. OpenAI's own docs list running evaluations, classifying large datasets, and embedding content as examples.

There's a middle ground too. Some work doesn't have anyone watching, but it needs to be done in minutes, not hours, like summarizing a meeting that just ended. That's a job for our own queue (https://www.kirupa.com/ai_fundamentals/async_processing_with_queues.htm) with workers making real-time calls. We control the pace, but we still pay real-time prices.

One quick note on words: batching also means something else in AI. When we run models on our own GPUs, grouping several requests into one pass through the model is also called batching, and it's about using the hardware efficiently. That's a cousin of this idea, known as request batching (https://www.kirupa.com/ai_fundamentals/request_batching.htm), and it has an article of its own.

Conclusion

Dog-Eared Books had two kinds of model work. The shopper's question needed an answer in seconds, so it went to the real-time API. The 40,000 summaries could wait until morning, so they went to a batch API, where, at the time of writing, the same work costs half as much and doesn't eat into the live assistant's rate limits.

The batch lane comes with its own habits: tag every request with a custom_id, match results by that ID, test a small sample first, and resubmit the lines that fail. Once those are in place, the question real-time or batch? mostly comes down to whether somebody is waiting on the other end.