AI glossary · Advanced · Efficiency & infra
Continuous batching
Continuous batching is a serving technique that groups multiple user requests to process them together on a GPU, removing finished requests right away and slotting in waiting requests right away so the GPU never sits idle.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-12
How it works
An LLM generates its answer one token at a time, and answer lengths differ from request to request. The older approach (static batching) accepted no new requests until the longest answer in the batch finished, even if slots freed up by requests that ended early sat empty. Continuous batching rebuilds the batch every time a token is generated. Finished requests return their results immediately, and waiting requests fill the empty slots. For this reason it is also called iteration-level scheduling.
Static batching vs continuous batching
Static batchingReplaced batch by batch
- Waits until the longest answer finishes
- Finished slots stay empty
- New requests wait for the next batch
Continuous batchingReplaced token by token
- Finished requests leave immediately
- Waiting requests join immediately
- Little GPU idle time
공통 Both compute multiple requests together to use the GPU efficiently
Why it matters
GPU idle time drops, so the same GPU can serve many more users. Higher throughput also lowers the cost of generating each token. Because new requests do not have to wait for the previous batch to finish, TTFT usually improves too. For these reasons, major inference engines such as vLLM, SGLang and TensorRT-LLM use it by default.
Good to know
You rarely turn it on or off directly; you encounter it through inference engine settings such as the maximum number of requests processed together or the maximum number of batched tokens. Setting these values high increases overall server throughput, but the token generation speed each user experiences can slow down. The more requests processed together, the more KV cache memory is needed, so it is paired with memory management techniques such as paged attention. When a very long prompt arrives, generation for other requests can be briefly delayed, so complementary techniques such as chunked prefill, which processes long inputs in pieces, are used alongside it.
Example
Suppose a team running a customer support chatbot gets complaints that responses lag every lunchtime. The team gradually raises the maximum number of concurrent requests in its vLLM settings while measuring server throughput, per-user response speed and TTFT together. When it finds the point where throughput keeps rising but the speed each user experiences falls below the target, it settles on the value just before that point and handles the remaining demand by adding another GPU.
Where it came up on this site
- Papers TokenRouter: An efficient serving system for token-level LLM routing 2026-10-10