AI glossary · Basic · MLOps & LLMOps · Efficiency & infra
Model serving
Model serving is the work of putting a trained AI model on servers and running it so that it receives requests from apps or users, runs the model and returns the results.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- MLOps & LLMOps, Efficiency & infra
- Added
- 2026-10-08
How it works
The serving system loads the model from a source such as a model registry and keeps it in the memory of hardware such as GPUs. When a user request comes in through an API, the server receives it, groups multiple requests or decides their order (batching and scheduling), runs the model and returns the inference result. Because LLMs generate answers one token at a time, streaming, which sends finished parts right away, is often used. Besides real-time responses, there is also batch processing, which handles accumulated data all at once.
Why it matters
No matter how good a model is, it cannot be used in a real service unless it is served. Serving determines response speed, how many requests can be handled at once, and cost. For LLMs in particular, GPUs are expensive, so techniques that handle more requests on the same GPU directly affect service costs. That is why optimization techniques such as KV cache management, quantization and speculative decoding are widely used at the serving stage.
Good to know
Serving is not something you start once and leave. It also requires scaling the number of servers up and down with request volume, deployment methods that keep the service running when switching to a new model, and failure monitoring. It often runs on Kubernetes, and during operation you also need to watch that performance does not drop because of data drift. Speed, cost and quality pull against each other, so you need to find the balance that fits the service's purpose.
Example
vLLM is an open-source engine widely used for LLM serving, and this site covered it in the news item 'vLLM v0.31.0 released, focused on optimizing DeepSeek-V4.1-Flash inference performance.' Companies use engines like this to run open-weight models on their own GPU servers and offer them as internal chatbots or APIs, and some place an LLM gateway in front to manage requests going to multiple models.