# Human Approval for Risky Actions

by [kirupa](https://www.kirupa.com/me/index.htm) | filed under [AI Fundamentals](https://www.kirupa.com/ai_fundamentals/index.htm)

When a new person joins a support team, they usually aren't handed the company credit card on day one. They can look things up and answer questions right away. Bigger things, like a large refund, go to a manager first. Nobody thinks the new hire is a bad person. It's just that some mistakes are cheap to undo and some aren't.

AI agents deserve the same arrangement. An agent with tools can do real things in the world: move money, send email, delete records. Most of the time it'll do the right thing. When it doesn't, we want a person to catch the mistake before it happens.

In the following sections, we'll sort an agent's tools by how risky they are, build an approval step that pauses the agent and waits for a person, and look at the details that decide whether that approval step actually protects anything.

Onwards!

## Tilly's Support Agent

Tilly runs an online shop called **Tilly's Teapots**, and she's built a support agent to handle customer emails. It has four tools: `get_order` looks up an order, `add_note` adds an internal note to a customer's account, `issue_refund` sends money back to a customer's card, and `send_email` emails the customer. This email just came in from a café owner:

> We ordered 16 teapots for our café (order #2207) and half of them arrived with chipped lids. We'd like a full refund of $640.

The agent looks up the order, sees the photos the customer attached, and decides a refund makes sense. Its next step is `issue_refund(2207, 640)`. Should that just happen?

## Not Every Tool Is Equally Risky

Before answering, let's sort the tools. A few questions help a lot here. Can the action be undone? Does it affect someone outside our company? How much money or data is involved? Here is how Tilly's tools land:

![Three bins of tools. Just do it, for read-only or easy-to-undo actions: get_order and add_note. Do it and log it, for small and easy-to-fix actions: issue_refund for 50 dollars or less. Ask a person, because money or words leave the building: issue_refund over 50 dollars, and send_email.](https://www.kirupa.com/ai_fundamentals/images/human_approval_for_risky_actions_risk_bins.png)

Looking up an order can't hurt anyone, so it runs freely. An internal note is easy to edit or delete. A small refund is real money, but if it's wrong, it's a small, recoverable mistake, so we let it run and log it. A large refund is much harder to claw back. An email to a customer can't be unsent at all, and it speaks for the company.

Tilly picked $50 as her line. Your line will depend on your business, and it's a business decision, so it's worth deciding with whoever owns the money. The $640 refund lands in the third bin, so it needs approval.

## Pause, Ask, Resume

Here is what the approval step does with that refund:

![Five steps. 1, the agent proposes issue_refund for order 2207 and 640 dollars. 2, our code checks and sees it is over 50 dollars. 3, the run pauses and its state is saved. 4, a person decides to approve, edit, or reject. 5, the run resumes and the refund is sent.](https://www.kirupa.com/ai_fundamentals/images/human_approval_for_risky_actions_pause_resume.png)

The agent proposes the [tool call](https://www.kirupa.com/ai_fundamentals/tool_calling.htm), just like it would for any other tool. It doesn't actually run yet.

Our code checks the call against Tilly's rules. This step is plain code, outside the model. It sees `issue_refund` with an amount over $50 and decides this one needs approval.

The run pauses, and its state gets saved somewhere durable, like a database. We save it because a person might not look at this for twenty minutes or three hours, and we don't want to keep a server process waiting that long or lose the whole run if something restarts. (This is the same idea as handing out a ticket in [async processing with queues](https://www.kirupa.com/ai_fundamentals/async_processing_with_queues.htm).)

A person sees an approval request and decides.

The run picks back up from where it paused. If the refund was approved, it runs, and the agent sees the result and carries on (probably to the email, which needs its own approval). If it was rejected, the agent sees that too, along with the reason.

## What the Approver Sees

The approval request is the whole safety net, so it needs to be good. ***The agent wants to issue a refund. Approve?*** is not enough information to make a real decision. Here's a better one:

![An approval card. Action: issue_refund. Order: number 2207, Café Rosa, 16 teapots. Amount: 640 dollars. Why: 8 of 16 lids chipped. Evidence: two photos. Buttons: Approve, Edit amount, and Reject. A handwritten note next to the Why row asks: 8 of 16?](https://www.kirupa.com/ai_fundamentals/images/human_approval_for_risky_actions_approval_card.png)

A good approval request shows the exact action and its exact arguments, the agent's reason in plain words, and the evidence it relied on. Here, the evidence raises a fair question. Only 8 of the 16 lids were chipped, so maybe the right refund is $320, or a set of replacement lids. That's why Edit amount is one of the buttons. The person can fix the amount instead of choosing between a wrong yes and an unhelpful no.

If the person rejects or edits, their note goes back to the agent. ***Refund half, and offer replacement lids*** gives the agent something useful to work with.

## The Gate Has to Be in Code

It's tempting to skip all of this and just put a line in the agent's instructions: ***Always ask a human before issuing refunds over $50***. The trouble is that instructions are suggestions. A model can misread them, forget them in a long conversation, or be talked out of them by text in the very email it's reading:

![Two panels. Rule in the prompt: the agent has been told to ask first, but an arrow goes straight from the agent to issue_refund, next to an email that says refunds are pre-approved. Gate in our code: an approval check sits between the agent and issue_refund, so the agent can only reach the tool through it.](https://www.kirupa.com/ai_fundamentals/images/human_approval_for_risky_actions_gate_in_code.png)

So the check has to live in code that wraps the tool, where the model can't reach it. The model can propose anything it likes. Only our code decides what actually runs.

There's a related detail that's easy to miss. The approval should be tied to the exact call the person saw. If someone approved `issue_refund(2207, 320)`, then that's the only call that runs. If the agent comes back with a different amount, or a different order, it needs a fresh approval. Otherwise ***approved*** turns into a blank check.

Here is how that plays out with two refund calls:

![Two tool calls. Someone approved issue_refund(2207, 320), so that exact call runs. If the agent comes back with a different amount, like issue_refund(2207, 640), it needs a fresh approval. Otherwise approved turns into a blank check.](https://www.kirupa.com/ai_fundamentals/images/human_approval_for_risky_actions_exact_call.png)

## Too Many Approvals Is Its Own Problem

There's a way to get this wrong in the other direction, too. If every single action needs approval, people get buried. After the fortieth ***Approve add_note?*** of the morning, they start clicking Approve without reading, and then the approval step isn't protecting anything.

Keeping the approval queue short is part of keeping it safe. Low-risk actions should run on their own. The ones that do need a person should come with enough context to decide quickly. If the same kind of request gets approved every time, that's a hint the threshold could move.

It's also worth deciding what happens when nobody answers. An approval request that sits for two days helps nobody, least of all the café owner. Give requests an expiry, tell the customer what's going on, and route old requests to someone who can deal with them.

Finally, write it all down: what the agent proposed, who approved or rejected it, any edits they made, and when. When someone asks ***why did we refund this café $320?***, that record has the answer.

## Conclusion

Tilly's agent can look things up, take notes, issue refunds, and email customers, and those tools carry very different amounts of risk. By sorting them, we let the safe ones run freely and stopped the risky ones at a gate. For the $640 refund, the agent proposed the call, our code paused the run and saved it, a person saw the exact action with its evidence, and they edited it down to a more sensible amount before anything happened.

What makes this hold up is where the pieces live. The gate sits in our code, where the model can't talk its way past it. Each approval covers exactly the call the person saw. And the queue stays short enough that people actually read what they're approving. With those in place, the agent can handle the everyday work, and the decisions that are hard to undo still go past a person first, the same way they would for a new member of Tilly's team.