AI glossary · Basic · Efficiency & infra · MLOps & LLMOps
Throughput (tokens/s)
Throughput (tokens/second) is the number of tokens an inference server produces per second, a metric that shows how much work the same hardware can handle. You need to distinguish between the total amount the whole server processes and the speed a single user experiences.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra, MLOps & LLMOps
- Added
- 2026-10-12
What it measures
Throughput is usually reported in two ways: the number of tokens the whole server produces per second across all requests (system throughput), and the number of output tokens a single request receives per second (per-user speed). As concurrent requests increase, continuous batching raises system throughput, but per-user speed gradually drops. The speed of reading input tokens and the speed of generating output tokens are also often measured separately, because input is processed all at once and is fast, while output is generated one token at a time and is slow.
Where to set the number of concurrent requests
- Real-time chatbot
- Maximum within TTFT target
- Overnight batch summarization
Where you see it and how to get a feel for it
It appears as tokens/s or tok/s in inference engine benchmark results, model pages from GPU cloud or API providers, and LLM API comparison sites. To roughly estimate the hardware your service needs, calculate 'tokens to process per day ÷ throughput of one GPU ≈ GPU hours needed'. Then estimate the cost per token as 'cost of one GPU hour ÷ tokens produced in that hour' and compare it with token-based API pricing. Output token generation is mostly bound by memory bandwidth, so memory bandwidth often affects perceived speed more than GPU compute performance (FLOPS).
Things to know
Throughput numbers vary widely with the number of concurrent requests, input and output lengths, precision and inference engine settings, so numbers measured under different conditions should not be compared with each other. Increasing batch size just to raise throughput can worsen TTFT and per-user speed. That is why, in practice, it is measured under a condition such as 'the maximum throughput achievable while TTFT stays below the target'. Different tokenizers also produce different token counts for the same sentence, which makes it hard to compare tokens/s directly across models.
Example
Suppose a company uses the same model for two jobs. An overnight batch job that summarizes tens of thousands of documents has no one waiting, so it sets a high number of concurrent requests to push system throughput to the maximum. For the chatbot employees use during the day, TTFT and per-user speed are met first, and then the number of GPUs needed is calculated from the throughput achieved under those conditions. Even with the same GPU, the settings and the number of GPUs needed differ by use case.