What is an AI gateway?

AI gateways started as a way for engineering teams to manage keys, costs and failover across model providers. They have become the place where AI governance meets live traffic. This guide explains what a gateway does, the actions it can take, how budgets and logging work, and what to look for in one.

Where the gateway sits

Without a gateway, every application and agent calls model providers directly, each with its own keys, its own retry logic and its own idea of what is allowed. Policy lives in documents and in whatever each team remembered to build.

With a gateway, traffic to models goes through a single layer. Applications call the gateway, the gateway applies the rules, and only then does the request reach the provider. The response comes back through the same layer, so it can be checked on the way out too. The gateway usually runs in your own environment, close to the applications, so prompts and responses do not have to leave your network to be inspected.

Every request carries an identity, usually which application or agent is calling, and the gateway uses that to decide which rules apply. That is why a gateway works best alongside an AI register. The register says what each system is and what it is approved to do, and the gateway enforces it.

What a gateway can do with a request

Each rule evaluates the request or response and returns a verdict. The common actions are these.

Allow. The request goes to the model unchanged. Most traffic should be allowed, and a gateway that blocks too much gets routed around.

Redact. Sensitive spans are found and masked or hashed before the request leaves, then the cleaned request goes through. Typical targets are names, contact details, government identifiers, card and bank numbers, health information and secrets such as API keys. Redaction keeps work flowing while keeping the data out of the model.

Block. The request is refused, with a message telling the caller why in neutral terms. Typical triggers are a prompt injection or jailbreak score above a threshold, a denied topic, a secret that must never leave, a model or tool that is not on the allowlist, or a caller that is not registered.

Reroute. The request is sent to a different model or provider, for example a self-hosted model for sensitive work or a fallback when the primary provider is down.

Warn. The request goes through but the event is flagged for review.

Escalate. The request is held or flagged for a person to decide, used where an action needs human approval.

Rules can also be run in observe-only mode, where the gateway records the verdict it would have reached without touching traffic. This is the safe way to introduce a new rule. Watch what it would have blocked for a week, tune it and then switch it on.

What a gateway checks

Gateways use scanners, which are small detectors that score or mark content. Common scanner types include the following.

• Personal information detectors that return the exact spans to redact.
• Prompt injection and jailbreak classifiers that return a score.
• Secrets detectors for keys, tokens and credentials.
• Toxicity and harmful content classifiers.
• Topic detectors for subjects a system must not discuss.

Beyond scanners, gateways enforce structural rules. A model allowlist restricts which providers and models a system may call. A good one also constrains failover, so an outage never routes traffic to a provider you have not approved. A tool allowlist restricts which tools an agent may call, either refusing the request or stripping the tool. A system-prompt check confirms the system prompt matches an approved version. Caching rules decide whether a response may be served from cache, and should never cache a response for a system whose traffic is redacted.

Many gateways can call out to external guardrail services or classifier models as well as their own built-in scanners, and let you write custom detectors for your own data.

Budgets and rate limits

AI cost scales with tokens, and a looping agent or a leaked key can burn through a month’s budget overnight. Gateways enforce two kinds of limit.

• A budget caps tokens or requests over a period, such as a million tokens a month for one system.
• A rate limit caps tokens or requests per minute.

Limits can apply to the whole organisation as a floor and to individual systems on top. When a limit is breached, the gateway either rejects the request before the provider is called or lets it through and records the breach. Counting on the gateway itself, rather than from the provider’s bill afterwards, is what makes the limit enforceable in real time. Unbounded consumption is one of the OWASP Top 10 for LLM Applications, covered in the red-teaming guide, so budgets are a security control as well as a cost one.

Logging and evidence

Every verdict the gateway reaches is an event. Blocks, redactions, reroutes, escalations and limit breaches should always be recorded. Allowed requests are usually sampled, because recording every one is expensive and rarely needed. Keep the event, not the full prompt, unless you have a reason and a legal basis to store content.

These events answer questions that otherwise have no answer. What did this system actually send to the model last month? How often did the redaction rule fire? Did anyone try to inject this agent? They feed monitoring, alerts and board reporting, and they can be exported to a SIEM, the security team’s central log and alerting system.

Logging also has a regulatory side. Under the EU AI Act, deployers of high-risk AI systems, meaning the organisations using them, must keep the logs those systems generate automatically, to the extent the logs are under their control, for at least six months unless other law says otherwise (Article 26(6)). A gateway is often the most practical place to hold the deployer’s side of that record. Set retention deliberately, balancing the evidence you need against privacy law on keeping personal information.

Fail open or fail closed

Sometimes a rule cannot be evaluated. A scanner times out, an external guardrail service is unreachable or the gateway has not received its rules. The gateway then has to decide what to do with the request. Fail closed refuses it. Fail open lets it through. Set this per system according to its risk, and record the choice, because it is a governance decision that an auditor may ask about.

How a gateway fits AI governance

The gateway turns parts of the governance framework into enforcement.

• The acceptable use policy becomes data rules and model allowlists.
• Risk assessment controls become specific guardrails for each system.
• Agent oversight becomes tool allowlists, budgets and escalation rules.
• Red-team findings become new scanner rules.
• Gateway events become monitoring measures, alerts and board figures.

For this to hold, rules need to be versioned and published like code. Draft, review, publish a locked version, deploy it to specific gateways and keep the history. A rule change that nobody can trace is a control nobody can rely on.

What to look for in a gateway

• Runs where your traffic runs, in your own environment, so prompts are inspected without leaving your network.
• Allow, redact, block, reroute, warn and escalate actions, plus observe-only mode.
• Span-level redaction, not only block or allow.
• Model and tool allowlists that also constrain failover.
• Budgets and rate limits counted on the gateway.
• Pluggable scanners, including external guardrail services and your own detectors.
• Per-system rules layered over an organisation-wide floor.
• Versioned, signed rule sets with a deployment history.
• Per-system fail mode that you set and record.
• Event logging with configurable sampling and retention, and export to your SIEM.

Where Aicura fits

Most of the checklist above describes Aicura’s gateway, which runs in your environment and pulls signed, versioned packages of guardrail rules, scanner scripts and budgets from Aicura. A global set is the floor for every system, a system’s own rule replaces the global rule targeting the same thing, and each registered system behind a gateway has its own recorded choice of failing open or failing closed. Rules can block, escalate, redact, reroute or warn, and any rule can run observe-only first.

Authoring guardrails starts at Essentials. Deploying to Aicura gateways, or exporting rule packages to a gateway you run yourself, is part of Pro, with personal information detection, LLM Guard, Presidio, Bedrock Guardrails in your own AWS account, Granite Guardian and your own scanner scripts. Enforcement events reach Aicura through the hub, the small Aicura service that links your network to the platform, with allowed requests sampled at the rate you set, and feed monitoring, alerts and KPIs.

A note on this page

This guide is general information about AI gateways as a category. It is not legal advice. Logging and retention choices involving personal information should be checked against the privacy law that applies to you.