Model Gateway for Keys, Quotas, and Routing

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

The first time we call a large language model from our app, it feels almost too easy. We paste in an API key, pick a model name, send a prompt, and get text back. Done!

Then a second feature wants a model. Then a third feature shows up, and that team prefers a different provider. A few months later, our API keys live in a handful of config files, nobody can say which feature ran up last week's bill, and trying a new model means changing code in several places. Nothing broke all at once. It just slowly got messy.

A model gateway is how we clean that up. It is a single service that sits between our apps and every model provider we use, and every model request goes through it. Because it sees every request, it is the natural place to keep the keys, enforce the quotas, and decide where each request goes. In the following sections, we'll walk one request through a gateway and look at each of those jobs up close.

Onwards!

Our Recipe App

To keep things concrete, imagine we are building a recipe app called Simmer. It has three features that call models. Chef Chat answers cooking questions while you're standing at the stove. Shrinker turns a rambling recipe blog post into a short list of steps. List Maker turns a week of meal plans into a grocery list.

Here is Chef Chat, waiting inside Simmer for the next cooking question:

[Image: Chef Chat sits in a chat panel in the corner of the Simmer recipe app. The panel header shows a robot avatar with a stew pot badge, the name Chef Chat, and the tagline Simmer's cooking helper. A cook asks, How do I keep pasta from sticking? Chef Chat replies, Use plenty of water, and stir it during the first couple of minutes! A handwritten note says This is Chef Chat!] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_meet_chef_chat.png)

Right now, each feature talks to its model provider directly, using its own copy of an API key:

[Image: Chef Chat, Shrinker, and List Maker each hold their own copy of a provider API key. Chef Chat and Shrinker call Provider A directly, and List Maker calls Provider B directly. There is no shared place for keys, limits, or logs.] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_before.png)

For one feature, this is fine. The trouble shows up as the number of features grows. Every copy of a key is one more place it can leak, and rotating a key means hunting down every copy. There's no single place to answer how much did List Maker spend yesterday? If Shrinker goes haywire, nothing stops it from burning through the Provider A rate limit that Chef Chat also depends on. And because each feature has its own code for its provider, switching models is a code change instead of a settings change.

All of these problems have the same shape. Every feature is making its own decisions about something that should be shared.

One Door In

What we are going to do is put one service in the middle. Our features stop talking to providers. They talk only to the gateway, and the gateway talks to the providers:

[Image: The same three features now send every request to one gateway in the middle. The keys have moved off the feature cards and into the gateway, which also tracks quotas and routes each request to Provider A or Provider B.] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_after.png)

A good way to picture this is the front desk of an office building. Visitors don't get a master key to every floor. They check in at the desk, the desk knows who they are and where they're allowed to go, and the desk keeps a log of who came and went. The gateway is that front desk for model requests.

A gateway usually speaks one request format to our apps, no matter which provider ends up answering. Many gateways pick an OpenAI-compatible format for this, since so many client libraries already speak it. Our features write their model code once, and the gateway does the translating.

Following One Request

Let's follow a single request from Chef Chat. Somebody types Can I swap butter for olive oil in brownies?, and Chef Chat sends that question to the gateway. Here is what happens inside:

[Image: A question from Chef Chat goes through five steps inside the gateway. 1, who are you: the key sk-gw-chef belongs to Chef Chat. 2, allowed: it is under its limits. 3, where to: chat-default means Provider A. 4, send it with the real provider key attached. 5, write it down: 612 tokens and 0.9 seconds.] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_steps.png)

In the first step, the gateway figures out who's asking. Chef Chat sent a key that starts with sk-gw-chef, but that isn't a real provider key. It is an internal key the gateway handed out earlier (often called a virtual key). The gateway looks it up, and now it knows this request belongs to Chef Chat.

In the second step, the gateway checks Chef Chat's rules. Is Chef Chat allowed to use this model? Is it under its request limit for this minute, and under its budget for this month? If the answer to any of those is no, the request stops right here.

In the third step, the gateway decides where the request goes. Chef Chat asked for a model called chat-default. That is a name we made up. The gateway's configuration says that chat-default currently means one particular model at Provider A.

In the fourth step, the gateway translates the request into Provider A's format, attaches the real Provider A key, and sends it off. The answer comes back through the gateway the same way.

In the fifth step, the gateway writes down what just happened: who asked, which model answered, how many tokens it used, what that cost, and how long it took. (The 612 tokens and 0.9 seconds are just example values.)

Those five steps map onto the three jobs in this article's title. Let's look at each one.

Keys

The real provider keys now live in exactly one place, the gateway (ideally in a secrets manager that the gateway reads from). Our features only ever hold their gateway keys.

This changes what a leak means. If someone accidentally commits Shrinker's key to a public repo, we revoke that one gateway key. Chef Chat and List Maker keep working, and the real provider key was never exposed at all. Rotating a real provider key gets easier too, because we change it in one place and none of the features notice.

Virtual keys also give every request an identity. Each request now carries a label saying which feature (or team, or customer) it came from. That label is what makes per-feature quotas and per-feature bills possible.

Quotas

Providers put rate limits on our account, usually counted in requests per minute and tokens per minute. Those limits are shared by everything that uses the account, and that sharing is where the trouble starts.

Imagine Shrinker ships a bug where a retry loop never stops. Without a gateway, Shrinker eats the entire Provider A rate limit, and Chef Chat starts getting 429 Too Many Requests errors from the provider. One broken feature takes down a feature that did nothing wrong. With a gateway, each feature gets its own slice of the limit:

[Image: Each feature has its own request limit. Chef Chat has used 120 of 300 requests this minute and List Maker has used 60 of 200. Shrinker has used all 200 of its requests, so it gets a 429 error. A handwritten note says only Shrinker waits.] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_quotas.png)

The gateway sends that 429 to Shrinker itself, before the request ever reaches the provider. Chef Chat never notices a thing.

Quotas usually come in two flavors. Rate limits (requests or tokens per minute) protect us from sudden bursts, like that runaway loop. Budgets (dollars per day or per month) protect us from slow leaks, like a feature that works perfectly but is far more popular than anyone planned for. Because the gateway records the token count of every response, it can turn tokens into dollars as requests flow through. (If the link between tokens and money still feels fuzzy, Tokens and How They Set Your Bill (https://www.kirupa.com/ai_fundamentals/tokens_and_your_bill.htm) walks through it.)

Deciding what happens when a budget runs out is a product decision. We could block the feature, switch it to a cheaper model, or keep going and page a human. The gateway just gives us one place to make that call.

Routing

Back in step three, Chef Chat asked for chat-default instead of a real model name. That one small indirection is what makes routing possible. Here is the kind of table the gateway uses to turn names into models:

The app asks for | The gateway sends it to
chat-default | A large model at Provider A, with 10% of traffic going to a new model we're trying out
summarize | A medium model at Provider B
cheap-fast | A small model at Provider B

Because our features only know the made-up name, we can change what's behind it without touching their code. That opens up some handy moves. We can try a new model on 10% of Chef Chat's traffic before switching everyone over. We can send easy requests to a cheaper model, which is what Routing Easy Tasks to Cheaper Models (https://www.kirupa.com/ai_fundamentals/routing_to_cheaper_models.htm) is all about. And if Provider A is having a bad day, the gateway can send the request to Provider B instead, which we cover in Fallback Models (https://www.kirupa.com/ai_fundamentals/fallback_models.htm).

One thing to keep in mind is that two models rarely behave identically, even with the same prompt. Swapping what's behind chat-default is a one-line configuration change, but it is also a behavior change for our users, so it deserves the same testing we'd give a code change.

What a Gateway Costs Us

A gateway isn't free. Every request now takes one extra network hop, which adds a little latency. The gateway is also a new piece of infrastructure, and if it goes down, every AI feature goes down with it. It needs to be run like the important service it now is.

The gateway also sees every prompt and every response. That makes its logs some of the most sensitive data we have, so we should decide early what gets logged, who can read it, and how long we keep it. Stripping personal information out of those logs (known as PII redaction (https://www.kirupa.com/ai_fundamentals/pii_redaction.htm)) has an article of its own.

Here are those costs together:

[Image: Three costs of a gateway. Every request takes one extra network hop, which adds a little latency. If the gateway goes down, every AI feature goes down with it. And it sees every prompt and response, so its logs are some of the most sensitive data we have.] (https://www.kirupa.com/ai_fundamentals/images/model_gateway_costs.png)

We don't have to build a gateway from scratch, either. There are open-source ones (LiteLLM's proxy is a popular choice), hosted ones like Cloudflare AI Gateway, and the big cloud platforms offer their own. A small internal service that does only the five steps we walked through is also a perfectly reasonable place to start.

And if we have one feature, one provider, and one developer? We probably don't need a gateway yet. The pain shows up around the second or third feature, which is a good time to add one, before the keys have spread too far.

Conclusion

We started with three features that each held their own keys and made their own choices. By putting a gateway in the middle, those shared decisions moved into one place. The gateway knows who's asking, so it can hand out keys and enforce quotas per feature. It owns the mapping from our made-up names to real models, so routing becomes a configuration change. And because it writes everything down, we finally have one place to answer who spent what?

The part I like most is how boring it makes the apps. Chef Chat asks for chat-default and gets an answer back. Providers, keys, limits, and logs are all the gateway's problem now. Next, we'll put that routing table to work by sending easy requests to cheaper models.