AI glossary · Basic · Efficiency & infra · MLOps & LLMOps
Inference engine
An inference engine is the runtime software that loads a trained AI model onto GPUs and lets it answer user requests quickly and efficiently. Widely used options include vLLM, SGLang and TensorRT-LLM for servers, and llama.cpp and Ollama for personal PCs.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra, MLOps & LLMOps
- Added
- 2026-10-12
How it works
An inference engine reads the model file (weights), loads it into GPU memory, and when a request comes in, generates tokens one at a time and returns them. It bundles techniques such as continuous batching, which processes multiple user requests together; paged attention, which manages the KV cache in blocks; fast attention computation (FlashAttention); running quantized models; and splitting a model across multiple GPUs. Most engines also start an OpenAI-compatible API server, so apps can keep using familiar API calls.
Where the inference engine sits
Apps · servicesChatbots, internal tools
API serverAccepts OpenAI-compatible requests
Inference engineBatching · KV cache · quantization · parallelism
Model weightssafetensors · GGUF files
GPUActual computation and memory
Where you'll see it
The usage and deployment sections of Hugging Face model cards often include run commands such as 'how to serve with vLLM'. Cloud model-serving services also run engines like these under the hood. In run commands and config files, you'll come across options such as maximum context length, GPU memory utilization ratio, tensor parallel size and quantization method. These values largely determine how many users can be served at once and how fast responses are.
Things to know
To serve many users, server engines such as vLLM, SGLang and TensorRT-LLM are common choices; for personal use on your own laptop, the llama.cpp and Ollama family, which run GGUF files, are common. Even for the same model, TTFT and throughput vary widely depending on the engine and settings, so when looking at performance numbers, check which engine and settings were used to measure them. Newly released models may not run, or may lack some features, until an engine adds support. When choosing an engine, also check that supported GPU types and quantization formats differ from engine to engine.
Example
If a team building an internal document chatbot picks an Instruct model from Hugging Face, they follow the instructions on the model card to start an OpenAI-compatible server with a single vLLM command, then adjust the maximum context length and GPU memory utilization ratio. A developer who just wants to try it out on a laptop, by contrast, downloads a GGUF file of the same model and runs it with Ollama. This site's news also carried a community report that Google AI Edge open-sourced 'ML Drift', an on-device GPU inference engine; engines for running models on phones and PCs are developing separately like this.
Where it came up on this site
- Papers TokenRouter: An efficient serving system for token-level LLM routing 2026-10-10
- briefing Conway Research launches invite-only beta of 'Underdog', a PC AI assistant that keeps data on the device (reported) 2026-10-09
- briefing Google AI Edge open-sources 'ML Drift', an on-device GPU inference engine (community report) 2026-10-09
- Papers STEPQuant: When and where errors matter in quantizing delta-rule recurrent states 2026-10-09
- Papers TRACE: Rollout-based QAT (quantization-aware training) for FP4 reinforcement learning of MoE language models 2026-10-08