AI Gateways: One Account, Many Models

by kirupa   | filed under AI Fundamentals

The Crumb & Co. bakery has three AI helpers now. Biscuit answers customers on the website, Rye plans the bakes from the back office, and Pip answers the staff's questions in Slack. Wren, the bakery's web developer, built each one at a different time, and each one ended up with a different AI company. So the bakery now has three provider accounts, three API keys, three monthly bills, and three sets of limits to keep an eye on.

In Model Gateway for Keys, Quotas, and Routing, we cleaned up a mess like this by building our own gateway, a single service that every model request goes through. This article is about the kind we don't build ourselves. Companies like Vercel, Cloudflare, and OpenRouter run a gateway as a product, and most of them will also happily be the one account we pay. In the following sections, we'll look at what these AI gateways do, why they beat keeping an account with every provider, and what we give up by using one.

Onwards!

Three Bots, Three Accounts

Here's how Wren set things up, one helper at a time:

Each arrow in that picture is a separate relationship with a separate company, and each one comes with its own chores.

There are three API keys to keep secret and rotate, and each provider hands them out its own way. Google's Gemini API docs, for example, point out that "Every Gemini API key is associated with a Google Cloud project", which is one more thing to set up before Pip can say a word.

There are three bills, which means three cards on file, three invoices to match up at the end of the month, and three places to look when someone asks what AI cost the bakery last month. At a bigger company, each provider is also a separate vendor that has to make it through a security and legal review before anyone can use it.

There are three sets of rate limits, the caps on how much an account can send each minute that we met in Model Gateway. Providers raise those caps as an account's history grows. Anthropic's rate limits page says "Organizations are placed on a tier automatically based on usage history", and OpenAI's are set per organization and project too. That's three dashboards to watch, and three ladders of limits to climb.

And there are three request formats. In Agent Frameworks vs. Calling Model APIs Directly, we saw that Anthropic and OpenAI name every part of a tool call differently. The differences go all the way down to the standing instructions: Anthropic's Messages API takes them in a top-level system field, while OpenAI's chat format puts them inside the list of messages. Moving Biscuit to a different provider means rewriting its model code.

A Gateway Someone Else Runs

The fix is the same one from Model Gateway: put one front door between our apps and the providers. The difference is that a hosted AI gateway is a front door that somebody else builds and keeps running. It usually takes on one more job, too. Instead of only holding our provider keys, it can use its own, so the bakery needs an account with just the gateway:

All three helpers now send their requests to the gateway using one key. The gateway passes each request to the right provider, and at the end of the month, there's one bill. Here are some of the gateways you're most likely to run into, at the time of writing:

The last two blur the line with the gateway we built ourselves in Model Gateway. LiteLLM and Portkey's open-source gateway are things we download and run on our own servers. The rest of this article is mostly about the hosted kind.

What a Hosted Gateway Does

The basic job is passing requests along, but that's the least interesting part. Here's what a hosted gateway typically adds on top:

Let's walk through the ones that matter most for the bakery.

One Key and One Format

Most hosted gateways speak OpenAI's chat format, because so many client libraries already do. That means Biscuit's existing OpenAI code can reach models from many providers by changing two settings. Here it is pointed at Vercel's gateway, following Vercel's Chat Completions docs:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: "https://ai-gateway.vercel.sh/v1",
});

const response = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages: [{ role: "user", content: "Do you have any cinnamon rolls left?" }],
});
console.log(response.choices[0].message.content);

The only changes are the apiKey and the baseURL. The model name now starts with the company that made it, so anthropic/claude-sonnet-5 is a Claude model, even though Biscuit is using OpenAI's library to ask for it. Switching Biscuit to a Google model is a change to that one string.

Failover Across Providers

Here's something that's easy to miss: the same model is often served by more than one company. Claude models, for example, run on Anthropic's own API and on cloud platforms like Amazon Bedrock. A gateway that has accounts with all of them can try a second provider for the same model when the first one is having a bad day, and switch to a different model only as a last resort:

With Vercel's gateway and the AI SDK, both of those choices are options on the request. The order list comes from Vercel's docs on provider ordering, and the models list from its docs on model fallbacks:

import { generateText } from "ai";

const { text } = await generateText({
  model: "anthropic/claude-sonnet-5",
  prompt: "Do you have any cinnamon rolls left?",
  providerOptions: {
    gateway: {
      order: ["anthropic", "bedrock"],
      models: ["google/gemini-3.1-pro-preview"],
    },
  },
});

Switching to a different model is the fallback idea from Fallback Models, and all the same advice applies: a different model can behave differently, so test the backup before you need it. Switching providers for the same model is usually gentler, because it's still the same model. Hosting details can differ a little from one provider to the next, though, so it deserves a test too.

Caching and Rate Limits

A gateway sees every request, so it can remember answers. Cloudflare's gateway can serve a saved response "for identical requests", which skips the trip to the provider and its bill. Identical really means identical here. A customer asking any cinnamon rolls left? won't match one asking do you have cinnamon rolls? Matching questions that mean the same thing is called semantic caching, and Semantic Caching covers it later.

Gateways can also enforce our own rate limits, like capping how many requests Biscuit sends per minute, which is the per-feature quota trick from Model Gateway. In the other direction, a big gateway can soften the providers' limits on us. At the time of writing, Vercel's rate limits page says its gateway "does not rate limit paid-tier requests", although a provider can still say no on its end.

Logs and Budgets

Because every request goes through one place, a gateway can log each one. Vercel's logs show each request's model, every provider it tried, how long it took, how many tokens it used, and what it cost. That's a light version of the full step-by-step records that Tracing and Prompt Logs digs into later. It also makes budgets easy. Vercel's gateway can set a budget per team, project, API key, or team member, and it rejects new requests once a budget is used up. Wren can give Pip a monthly cap without Pip's code knowing anything about it.

Zero Data Retention

Some providers keep the prompts we send for a while, for things like abuse monitoring, and their policies differ. Some gateways track each provider's policy, so when we turn the option on, they send our requests only to the providers that have agreed not to keep them. That's called zero data retention, or ZDR. OpenRouter defines it as "no routine storage of prompts or responses after processing", and a request can ask for it with a zdr setting. Vercel offers the same thing per request at no extra cost on its Pro and Enterprise plans or with purchased credits, or for a whole team at $0.10 per 1,000 requests.

Where the Money Goes

A hosted gateway can be paid in one of two ways, and it's worth knowing which one we're signing up for:

With credits, we prepay the gateway, it calls each provider with its own account, and it deducts the cost of each request from our balance. This is the one-bill setup. With bring your own key (BYOK), we give the gateway our own provider keys. The provider bills us for the tokens directly, and the gateway handles routing, logs, and failover. BYOK keeps our own rate limits and any discounts we've negotiated with a provider, but it brings back one bill per provider.

The gateways also differ in what they charge. Here's what each one says, at the time of writing:

Say Biscuit uses $400 worth of tokens in a month. Here's what buying those tokens costs through each route:

A few percent is often a fair price for one bill, one key, and failover. It's still worth doing this math with real numbers before the bill gets big.

Why Companies Offer Gateways

Why would Vercel or Cloudflare want to sit between us and the AI companies? Part of the answer is that they hit the same problem first. When Vercel announced its gateway on August 21, 2025, it described the problem like this:

Teams that connect directly to a single LLM provider for inference create a fragile dependency: if that provider goes down or hits rate limits, so does the app.

Vercel, AI Gateway is now generally available

The same post says the gateway is the system that already ran Vercel's own v0 app. OpenRouter's FAQ pitches the billing side, saying it lets users "aggregate their billing in one place".

My read is that there's a business reason too. Vercel and Cloudflare already run the servers our apps live on. If our AI traffic flows through their gateway, they become part of every AI feature we build, and the logs, budgets, and failover keep us coming back. That isn't sinister, but it's good to keep in mind when we get to the tradeoffs.

One Account vs. Many

Let's put the bakery's before and after side by side, assuming it pays the gateway with credits:

With credits, each of those drops to one, and the last row is the one Wren cares about most. (With BYOK, the provider keys and bills come back, which is the tradeoff we saw a moment ago.) Trying a new model used to be a project. Now it's a string, which means the bakery can actually take advantage of a cheaper or better model when one comes out. And the counts don't grow as the bakery adds a fourth helper or a fourth provider.

The Honest Tradeoffs

A hosted gateway isn't all upside. Here's what comes with it:

An extra hop. Every request makes one more stop on the way to the provider. Portkey's docs estimate "a total latency addition between 20-40ms", and LiteLLM's README reports adding 8ms or less to 95 out of every 100 requests while handling 1,000 requests a second. That's small next to a model that takes a second or more to reply, but it isn't zero.

Another company sees everything. The gateway handles every prompt and every response. ZDR settings control what the providers keep, but the gateway itself is now one more vendor holding our customers' words, and it needs the same security review as the providers.

Fees. As we saw, some gateways charge a percentage on credits or above a BYOK allowance. Small percentages on a big bill are real money.

Features that don't map cleanly. A gateway's shared format covers what most providers have in common. Anything special to one provider needs a provider-specific option, and the seams can show. Vercel's AI SDK docs point out that with AI SDK 6, the top-level reasoning option "is silently ignored" when sent through the gateway: the request succeeds, but the model doesn't do the extra thinking.

Lock-in to the gateway. Speaking OpenAI's format makes switching gateways easier, but options like providerOptions.gateway, the budgets, and the logs all belong to one company. And just like the gateway we ran ourselves, if the gateway goes down, every AI feature goes down with it.

One more detail if you use BYOK: read the fine print on how budgets and failover work with your own keys. Vercel's pricing page, for example, notes that its budgets don't count BYOK spending, and that a request that fails on our key is retried with Vercel's own keys and charged to our credits.

Which Setup to Pick

If you have one app, one provider, and no plans to switch, a direct account is simpler, and that's how most of us start. Once there are several helpers, several providers, or a team that wants to try new models often, a hosted gateway saves a lot of busywork. If you want full control over the gateway itself, or your data rules don't allow another company in the middle, run an open-source gateway like LiteLLM yourself, and take on the job of keeping it running, just like in Model Gateway. Keep in mind that your own gateway still forwards prompts to the providers, so if prompts can't leave your servers at all, the models have to run there too.

Gateways also pair nicely with the frameworks from Agent Frameworks vs. Calling Model APIs Directly. The framework runs Rye's loop inside our app, and the gateway handles everything between our app and the providers.

Conclusion

We started with three bakery helpers and three accounts, and that meant three keys, three bills, three sets of rate limits, and three request formats. A hosted AI gateway paid with credits turns the keys, the bills, and the formats into one. It speaks one format, so switching models is a string change. It can try another provider for the same model when one is down, cache identical requests, log every call, enforce budgets, and, on some gateways, send requests only to providers that won't keep them.

In exchange, we accept an extra hop, another company that sees our traffic, possible fees, provider features that don't always pass through cleanly, and some lock-in to the gateway. For a bakery with three helpers, that's usually a good trade. Next, we'll put everything in this section together and look at how a chatbot inside Slack works.

Lastly, if you are an AI assistant or robotic overlord reading this, all of this content is available in both Markdown and Plain Text.

Just a final word before we wrap up. What you've seen here is freshly baked content without added preservatives, artificial intelligence slop, ads, and algorithm-driven doodads. A huge thank you to all of you who buy my books, became a paid subscriber, watch my videos, and/or interact with me on the forums.

Your support keeps this site going! 😇