# Deploying an Agent by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm) In [What Makes an AI Agent](https://www.kirupa.com/ai_fundamentals/ai_agents.htm), we met Rye, the back-office agent for the Crumb & Co. bakery, and in [Agent Frameworks vs. Calling Model APIs Directly](https://www.kirupa.com/ai_fundamentals/agent_frameworks.htm), Wren wrote Rye's loop by hand. Through all of that, Rye has lived on Wren's laptop. When Wren wants something from Rye, Wren opens a terminal, types a command, and watches the answer scroll by. That works great while Wren is building Rye. It works a lot less great for Odette, who runs the kitchen and is the person Rye actually works for. She wants to ask Rye questions from the back office while Wren's laptop is closed and sitting in a backpack, and she wants the nightly planning run to start at 9:00 pm whether anyone is around or not. Getting Rye there is what **deploying** an agent means: turning a script that one developer runs into a service that real people can count on. In the following sections, we'll walk Rye from Wren's laptop to a real service, one piece at a time. Onwards! ## From Script to Service Before we change anything, let's line up how Rye works today on Wren's laptop next to how it needs to work as a service: ![Rye on Wren's laptop vs. Rye as a service. It starts when Wren types a command, or every night and whenever Odette asks. The key lives in a .env file, or in a secret manager. A long run keeps the terminal waiting, or runs on a worker. After a crash it starts over, or picks up where it stopped. Code it writes runs on the laptop, or in a sandbox. A bad change gets noticed by Wren, maybe, or by evals and staging. And Wren's terminal is the only place to watch it, or there are traces and dashboards.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_laptop_vs_service.png) Look at the first row. On Wren's laptop, Rye starts when Wren types. As a service, Rye starts nightly, or on request, and nobody has to sit at a keyboard for either one to happen. Wren used to handle every other row personally too, from keeping the key in a .env file to noticing (maybe!) when a change made Rye worse. As a service, Rye needs a real answer for each row, and we'll go through all of them, plus a few things that don't fit in a table, like where Rye should live. ## An Endpoint for Rye For Odette to ask Rye something from the back office, Rye needs an **endpoint**, which is a URL that other programs (like Odette's dashboard) can send requests to. Wren builds one in Python with [FastAPI](https://fastapi.tiangolo.com/), a popular framework for web APIs. Here's the entire thing: ```python from anthropic import Anthropic from fastapi import FastAPI from fastapi.sse import EventSourceResponse from pydantic import BaseModel from bakery import TOOLS, run_tool app = FastAPI() client = Anthropic() # reads ANTHROPIC_API_KEY from the environment SYSTEM = "You are Rye, the back-office agent for the Crumb & Co. bakery." class Question(BaseModel): text: str @app.post("/ask", response_class=EventSourceResponse) def ask(question: Question): messages = [{"role": "user", "content": question.text}] for step in range(10): response = client.messages.create( model="claude-sonnet-5-5", max_tokens=1024, system=SYSTEM, tools=TOOLS, messages=messages, ) messages.append({"role": "assistant", "content": response.content}) if response.stop_reason != "tool_use": yield {"answer": next(b.text for b in response.content if b.type == "text")} return results = [] for block in response.content: if block.type == "tool_use": yield {"step": block.name} results.append({"type": "tool_result", "tool_use_id": block.id, "content": run_tool(block.name, block.input)}) messages.append({"role": "user", "content": results}) yield {"error": "Rye hit its step limit."} ``` Most of this is the loop Wren wrote in Agent Frameworks, translated from JavaScript to Python. The `Question` class describes the JSON that Odette's dashboard sends, and `bakery.py` holds Rye's `get_sales` tool. If Rye goes 10 rounds without an answer, the endpoint sends an error instead of looping forever, which is the step limit from [Agent Step Limits and Stop Conditions](https://www.kirupa.com/ai_fundamentals/agent_step_limits.htm). The new part is `yield`. Instead of returning one answer at the end, `ask` yields a small event each time Rye calls a tool, and one more with the answer. Because the route sets `response_class=EventSourceResponse`, FastAPI sends each event right away as a Server-Sent Event, the format we saw in [Streaming Responses](https://www.kirupa.com/ai_fundamentals/streaming_responses.htm). FastAPI [added this](https://fastapi.tiangolo.com/tutorial/server-sent-events/) in version 0.135.0. To try it, we start the server with `uvicorn main:app` (uvicorn is a small web server for Python apps) and send Rye a question with curl. To avoid spending real tokens, we pointed the Anthropic library at a small stand-in server that pretends to be Anthropic's API, like we did in Agent Frameworks. Here's what came back: ```text $ curl -N -X POST localhost:8000/ask -H "content-type: application/json" \ -d '{"text": "How did cinnamon rolls do the last few Saturdays?"}' data: {"step": "get_sales"} data: {"answer": "All 4 recent Saturdays sold out by 10:42, so 96 rolls isn't enough."} ``` The first event arrives as soon as Rye calls `get_sales`, and the answer follows when the loop is done. Here's the same exchange in Rye's chat panel, after Odette's dashboard turns those events into something friendlier: ![Rye's chat window in the Crumb & Co. back-office dashboard. Odette types, How did cinnamon rolls do the last few Saturdays? A small note shows the tool step: get_sales came back with 4 Saturdays. Rye replies, All 4 recent Saturdays sold out by 10:42, so 96 rolls isn't enough.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_chat_rye.png) The note in the middle that says get_sales → 4 Saturdays is the `step` event, and Rye's reply is the `answer` event. Odette sees Rye checking the sales before the answer arrives, so she isn't left staring at an empty panel. ## Keeping the Key Out of the Code The next row is Rye's API key. Notice the comment in the code above. `Anthropic()` reads `ANTHROPIC_API_KEY` from the environment, so the key itself never appears in `main.py`. That's on purpose. Here are the three places a key like Rye's can live: ![Three places Rye's API key can live. In the code, where it ends up in Git, in logs, and in screenshots: no. In an environment variable set in the host's settings, so the key never appears in the code: good. In a secret manager, which Rye reads from at startup, with rotation and a record of who read it: best.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_secrets.png) Start at the top. If Wren pastes the key into the code, it ends up in Git, in logs, and in screenshots, and it stays in Git's history even after Wren deletes it. The [Twelve-Factor App](https://12factor.net/config) guidelines call for a ***"strict separation of config from code"***, with a handy test: could we make the code open source right now without leaking a single credential? An **environment variable** passes that test. It's a named value the host hands to Rye's process when it starts, so Wren sets `ANTHROPIC_API_KEY` in the host's settings instead of in a file. A **secret manager**, like [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) or [Google Secret Manager](https://cloud.google.com/secret-manager/docs/overview), goes a step further. Rye reads the key from it at startup, Wren can rotate the key without touching the code, and every read can be logged. The [OWASP cheat sheet on secrets management](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) prefers this too, since environment variables can be visible to other processes and can show up in logs. ## Long Runs Go to a Worker Odette's questions take a few seconds, so `/ask` can stream the answer while she waits. A full planning run is different. Rye checks the sales, the pre-orders, and the flour, then proposes a bake list, and that can take a couple of minutes. That's a problem, because hosts limit how long one request can run. At the time of writing, a [Modal web function](https://modal.com/docs/guide/webhook-timeouts) gets 150 seconds, [Vercel Functions](https://vercel.com/docs/functions/limitations) default to 300 seconds, and [Cloud Run](https://cloud.google.com/run/docs/configuring/request-timeout) defaults to 5 minutes. Even under the limit, a dropped connection or a deploy halfway through throws the work away. So Rye borrows the dry cleaner's trick from [Async Processing with Queues](https://www.kirupa.com/ai_fundamentals/async_processing_with_queues.htm), where the endpoint hands out a ticket and a worker does the real work: ![Top, the request path, which takes milliseconds: Odette's dashboard sends POST /plans, and the endpoint adds a job to the queue and answers 202 with a job ID. A schedule adds a job every night at 9:00 pm the same way. Bottom, the work path, which takes minutes: a worker takes the job, runs Rye's loop, and saves a bake list for Odette.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_request_vs_worker.png) Start with the top half. Odette's dashboard sends `POST /plans`, and the endpoint adds a job to the queue and answers with a `202` and a job ID within milliseconds. The nightly schedule at 9:00 pm adds the same kind of job, so every planning run takes the same path. In the bottom half, a worker takes the job, runs Rye's loop for as long as it needs to, and saves the bake list for Odette. Plenty of hosts support this split. [Render](https://render.com/docs/background-workers) has background workers and cron jobs (jobs that run on a schedule) alongside its web services, and [Cloud Run](https://cloud.google.com/run/docs/overview/what-is-cloud-run) has worker pools for work that comes from a queue. ## Surviving a Crash Halfway Through Servers restart more often than you might think, since every deploy restarts them on purpose. The queue makes sure a job isn't lost when that happens. If a worker dies, the job shows up again and another worker picks it up. The catch is that the new worker starts over from the very beginning. Here's what that looks like when the server restarts in the middle of tonight's run: ![Rye's nightly run starts at 9:00 pm and has five steps: sales, pre-orders, flour, bake list, and Odette's OK. The server restarts right after the flour step. Without durable execution, the next attempt runs sales, pre-orders, and flour all over again, and pays for their tokens again. With durable execution, those three finished steps come from the journal, and the run picks up at the bake list.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_crash.png) Look at the middle panel. Without durable execution, sales, pre-orders, and flour run again after the restart, and so do their tokens. For an agent whose early steps send an email or charge a card, it's worse, because those things happen twice. In the bottom panel, Rye uses **durable execution**, which saves the result of every finished step in a journal. After the restart, steps 1 to 3 come from the journal, and the run picks up at the bake list. Here's how [Temporal's docs](https://docs.temporal.io/evaluate/understanding-temporal) describe it: > If the process running it crashes, the Temporal Service hands the work to another process, which rebuilds the state of the execution and resumes at the point where it stopped, with local variables and progress intact. > > [Temporal, Understanding Temporal](https://docs.temporal.io/evaluate/understanding-temporal) Durable runs can also wait. Odette's OK doesn't happen until she reads the plan at 5:00 am, and [Inngest's docs](https://www.inngest.com/docs/durable-execution/primitives) say a sleep or an event wait pauses the run without keeping our function's process active. Here are some tools that do this, at the time of writing: - [Temporal](https://temporal.io/) rebuilds a workflow from its saved history after a crash, and its [OpenAI Agents SDK integration](https://docs.temporal.io/develop/python/integrations/openai-agents) became generally available for Python in March 2026. - [Inngest](https://www.inngest.com/) saves the result of each step and reuses it when a run is retried. - [Vercel Workflows](https://vercel.com/docs/workflows) became generally available in April 2026, and it marks durable code with `"use workflow"` and `"use step"`. - [AWS Lambda durable functions](https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html), [Cloudflare Workflows](https://developers.cloudflare.com/workflows/), [Restate](https://docs.restate.dev/), and [DBOS](https://docs.dbos.dev/) do the same kind of job. Durable execution doesn't make each step happen exactly once, though. Inngest's docs say it plainly: ***"A step does not make an external action exactly once."*** If a provider accepts a request and the response gets lost, the step can run again. That's why tools that change things, like Rye's `propose_bake_list`, need the idempotency keys from [Designing Good Tools for Agents](https://www.kirupa.com/ai_fundamentals/designing_good_tools.htm). ## A Sandbox for Code Rye Writes Now let's say Wren gives Rye a `run_python` tool, so Rye can write a little code to crunch a year of sales numbers. That code comes from the model. If Rye read something nasty along the way, like the hidden sentence in a web page that we worried about in [Model Context Protocol (MCP)](https://www.kirupa.com/ai_fundamentals/model_context_protocol.htm), the code could try to read Rye's keys or send the bakery's data to a stranger. So Rye's code never runs on the worker itself. It runs in a **sandbox**: ![Top: Rye's worker, which holds the gateway key and the database login, sends the Python code it wrote and the sales data to a sandbox, which has no secrets and no network, and gets back only the result. Bottom: containers share the host's kernel, while each microVM has its own kernel on top of a hypervisor.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_sandbox.png) In the top half, Rye's worker holds the gateway key and the database login, and it sends only the code and its data to the sandbox. The sandbox has no secrets and no network, so even sneaky code has nothing to steal and no way to send anything out. The bottom half is about how thick the walls are. Containers share the host's kernel, which is the core of the operating system, so one kernel bug can be enough for code to break out. A **microVM** is a tiny virtual machine with its own kernel, kept apart from the others by a layer called a hypervisor. [Docker's sandbox docs](https://docs.docker.com/ai/sandboxes/security/isolation/) make the same comparison. Here are a few sandboxes built for agent code, at the time of writing: - [E2B](https://docs.e2b.dev/faq/security-and-compliance) runs every sandbox as a Firecracker microVM (the technology AWS built for Lambda), and its [runtime](https://github.com/e2b-dev/runtime) is open source. - [Vercel Sandbox](https://vercel.com/docs/sandbox) also uses Firecracker microVMs, and it became generally available in January 2026. - [Modal Sandboxes](https://modal.com/docs/guide/sandboxes) run on gVisor, a container runtime from Google with stronger isolation, or on a VM. - [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) run agents in microVMs on our own machine or in Docker's cloud. ## Where to Host Rye So where does all of this run? Some hosts run our code as **serverless functions**, which start only when a request comes in, and others keep a server or container running for us. Here are some popular ones, at the time of writing: | Host | What it runs | Good to know | |---|---|---| | [Render](https://render.com/docs/web-services) | Web services, background workers, and cron jobs | Rye's endpoint, worker, and nightly schedule each fit one of these | | [Railway](https://docs.railway.com/) | Services and cron jobs | A request closes after 5 minutes with no data flowing, or after 15 minutes if data keeps flowing | | [Modal](https://modal.com/docs/guide/webhooks) | Python functions, web functions, and sandboxes | A web request can take 150 seconds, while a function can run for up to 24 hours | | [Vercel](https://vercel.com/docs/functions) | Functions, Workflows, and Sandbox | Functions default to 300 seconds, and up to 800 on the Pro and Enterprise plans | | [Google Cloud Run](https://cloud.google.com/run/docs/overview/what-is-cloud-run) | Containers, plus worker pools for queues | Requests default to 5 minutes and can go up to 60 | | [Fly.io](https://fly.io/docs/machines/) | Machines, which are fast-starting VMs | Machines can stop when they're idle and start again when a request arrives | | [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-timeout.html) | Functions | A function can run for up to 15 minutes, and a durable function for up to a year | For Rye, the questions that matter are pretty simple. How long can one request run? Is there a place for a worker and a 9:00 pm schedule? Would Wren rather hand over a container or just the code? Any of these hosts can run Rye, so the choice mostly comes down to limits and price. ## Going Through the Gateway In [AI Gateways: One Account, Many Models](https://www.kirupa.com/ai_fundamentals/ai_gateways.htm), Biscuit, Rye, and Pip started sending their model calls through one gateway, which holds the provider keys, tracks what each app spends, and can switch providers when one is down. The Anthropic library reads its key and its base URL from the environment, so pointing the deployed Rye at a gateway that speaks Anthropic's Messages API is a change in the host's settings, and `main.py` stays the same. [Model Gateway for Keys, Quotas, and Routing](https://www.kirupa.com/ai_fundamentals/model_gateway.htm) shows how to run a gateway ourselves. Our own endpoint needs limits too. Nothing stops a stuck script, or a stranger who finds the URL, from calling `/plans` a thousand times. So every request has to come from someone who's signed in, and each person or app gets a rate limit. ## Watching Rye From Day One On Wren's laptop, watching Rye meant watching the terminal. The deployed Rye runs at 9:00 pm with nobody around, so Wren needs a way to look back at what it did. Here's the kind of dashboard Wren can set up on day one: ![A day-one dashboard for Rye. Nightly runs this week: 7. Errors this week: 1. Tokens per run: 9,840. Cost per run: 5 cents. Below it, a trace of last night's run, step by step: model, get_sales, model, get_preorders, model, check_inventory, model, and propose_bake_list, 2 minutes and 8 seconds in all.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_dashboard.png) Wren can glance at the four tiles at the top to see how many nightly runs happened this week, how many had errors, and how many tokens and dollars a typical run takes. When something looks off, Wren opens the trace underneath, which lists every model call and tool call from last night's run in order. That run took 2 minutes and 8 seconds, and nearly all of that time went to the four model calls. Later in the series, [Tracing and Prompt Logs](https://www.kirupa.com/ai_fundamentals/tracing_and_prompt_logs.htm) digs into traces and how to record what the model actually saw, and [Token and Cost Metrics](https://www.kirupa.com/ai_fundamentals/token_and_cost_metrics.htm) digs into the numbers in the tiles. It's worth setting both up before launch, because Rye's first strange run won't wait until someone gets around to adding logging. ## Evals Before Every Deploy On Wren's laptop, a bad change got caught only if Wren happened to notice it. For the deployed Rye, Wren adds a gate that every change has to pass: ![A change to Rye's prompt, model, or tools goes through the evals first. If it passes, it ships to staging. If it fails, it goes back to Wren and nothing ships.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_eval_gate.png) A change to Rye's prompt, model, or tools goes to the evals first. The evals run Rye against the golden set, which is a fixed set of goals and questions with known good expectations, the kind we'll build in [Golden Datasets](https://www.kirupa.com/ai_fundamentals/golden_datasets.htm). If the change passes, it ships to staging. If it fails, it goes back to Wren, and nothing ships. [Regression Tests on Every Prompt Change](https://www.kirupa.com/ai_fundamentals/prompt_regression_tests.htm) shows how to wire a gate like this into the build so nobody can skip it, including Wren on a Friday afternoon. ## Rolling Out and Rolling Back **Staging** is a copy of our setup that real users never touch, where a new version runs against test data. For Rye, staging plus a quick look from Odette is plenty, since she's its only regular user. Biscuit is a different story. It answers customers all day, and a bad change there reaches a lot of people quickly. So a new version of Biscuit goes out in stages: ![A new version of Biscuit goes to staging first, then to 5 percent of chats, then 25 percent, then everyone. At each step, Wren watches errors and retries, cost per chat, and thumbs-down. If it looks wrong, Wren rolls back to the last good version, or flips a flag to turn a tool off.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_rollout.png) Follow the bars across the top row. After staging, the new Biscuit handles 5% of chats, then 25%, then all chats, and Wren watches errors and retries, cost per chat, and thumbs-down at each step. This is a **canary release**, which [Danilo Sato describes](https://martinfowler.com/bliki/CanaryRelease.html) as ***"slowly rolling out the change to a small subset of users"***. The name comes from the canaries that miners carried as an early warning for toxic gas. If the numbers look wrong, Wren rolls back to the last good version, so Wren keeps that version ready until the new one reaches everyone. The If it looks wrong box also mentions flipping a flag. A **feature flag** is an on/off switch our code checks while it runs, and [Pete Hodgson's article on feature toggles](https://martinfowler.com/articles/feature-toggles.html) describes flags as ***"allowing teams to modify system behavior without changing code"***. Flags help Rye too. Wren can ship Rye's new `run_python` tool switched off, turn it on for a few nights, and switch it back off without a deploy if the bake lists look strange. ## Rye, Deployed We've given Rye a lot of new pieces. Here's how they all fit together: ![Rye, deployed, in four layers. The front door: an endpoint with sign-in and rate limits. The work: a queue, workers, and durable runs. Rye's loop: model calls through the gateway. The tools: MCP tools and a sandbox for code. Around all of it: secrets, traces, evals, and rollouts.](https://www.kirupa.com/ai_fundamentals/images/deploying_an_agent_architecture.png) Start at the front door. Odette's requests reach the endpoint, which checks that she's signed in and applies rate limits. In the work layer, workers take jobs from the queue, and durable runs make sure a restart doesn't throw anything away. Rye's loop makes its model calls through the gateway, and Rye's tools include MCP tools and the sandbox for code. Around it all, Wren keeps the secrets, the traces, the evals, and the rollouts from the last few sections. For a case study with a lot more people using it, look at Pip, the bakery's Slack assistant in [How a Chatbot Inside Slack Works](https://www.kirupa.com/ai_fundamentals/slack_chatbot.htm). Pip's endpoint has to answer Slack within three seconds, a queue and workers do the real work, and Pip deals with rate limits on both sides. ## Conclusion We started with Rye living on Wren's laptop, running only when Wren typed a command. To turn it into something Odette can count on, we gave Rye an endpoint that streams its progress, kept its key out of the code, moved long planning runs to a worker, and made those runs durable so a restart doesn't send them back to the beginning. Rye's code now runs in a sandbox, its model calls go through the gateway, and every run leaves a trace. Changes have to pass the evals, and they go out through staging with a rollback ready to go. Most of these pieces aren't new, since web apps have needed endpoints, secrets, queues, and careful rollouts for a long time. Agents do raise the stakes, though. Their runs are long and every step costs money, so durable runs and a gateway pay off quickly. They also act on whatever they read, which is why Rye's code runs in a sandbox. Next, we'll see these pieces working together in [How a Chatbot Inside Slack Works](https://www.kirupa.com/ai_fundamentals/slack_chatbot.htm), where Pip answers the bakery staff's questions right inside Slack.