AI glossary · Advanced · AI platforms & solutions · MLOps & LLMOps
LLM gateway
An LLM gateway is an intermediate layer that places a single entry point between multiple apps and the multiple AI models they use, routing every request through it. This entry point handles authentication, model selection (routing), cost management, logging and security checks in one place.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Added
- 2026-10-08
How it works
Apps such as internal chatbots, coding tools and workflow automation send requests to a single gateway address instead of calling each model provider's API separately. The gateway checks whether the requesting person or team has permission, then forwards the request to an external model or a model served in-house according to predefined rules. When the response comes back, it returns it to the app unchanged while logging the number of tokens used, the time taken and the cost. Many gateways convert multiple providers' APIs into one common format, so models can be switched without major changes to app code.
Why it matters
As more teams inside a company use AI, API keys end up scattered everywhere, and it becomes hard to track who is using how much and what data is leaving the company. A gateway lets you handle key management, per-team usage limits and cost allocation in one place. You can also set up fallbacks that automatically route requests to another model when one model slows down or goes down. Guardrails that mask personal information or block risky requests, and caching that reuses answers to identical questions, also fit well at this point.
Things to know
Because every request passes through one point, a slow or stalled gateway affects the whole service. That is why it needs to run on multiple instances with continuous health monitoring. Fitting many models into a single format can also make it hard to use features unique to a particular model right away. In addition, prompt and response logs accumulate in the gateway, so these logs should themselves be treated as sensitive data, with defined access permissions and retention periods.
Example
In companies where several departments build their own generative AI features, a platform team typically sets up a single LLM gateway and requires every team to access models only through it. For example, rules might send a customer support chatbot to a fast, cheap model and a contract review tool to a high-performance model, and at the end of the month the gateway logs are used to split usage and costs by department. Open-source gateways and managed AI gateway services from cloud providers are used for this purpose.