AI glossary · Basic · Efficiency & infra · MLOps & LLMOps
TTFT
TTFT (Time To First Token) is the time from when a user sends a request until the AI outputs the first token (a piece of text) of its answer. It is a key metric for making a chatbot feel 'responsive.'
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra, MLOps & LLMOps
- Added
- 2026-10-12
How it works
An LLM response has two phases. First comes the prefill phase, which reads the entire input prompt at once and builds the KV cache, followed by the decode phase, which generates tokens one at a time. TTFT is the sum of the time spent waiting in the server queue, network time and prefill time. So the longer the prompt and the busier the server, the higher the TTFT.
Where to see it and how to get a feel for it
You see TTFT in inference engine benchmark results, LLM API comparison sites and the dashboards of LLM observability tools. It is usually given in seconds or milliseconds, and looking at percentiles such as p50, p95 and p99 (focused on the slower requests) matches practical experience better than the average. Metrics viewed alongside it include time per output token (TPOT) and total response time. A rough rule of thumb is 'total response time ≈ TTFT + number of output tokens × time per token.'
Things to know
Reasoning models can spend a long time generating thinking tokens before the answer shown on screen, so the time users actually wait can feel longer than the TTFT figure. Attaching long documents to the prompt, as in RAG, lengthens prefill and increases TTFT, so prompt caching helps a lot. Using large batches to raise throughput can worsen TTFT, so the two need to be balanced. If streaming is not turned on, users see a blank screen until the whole answer is finished, no matter how short the TTFT is.
Example
Suppose a team choosing a model for a customer support chatbot compares two candidate APIs. The team compares p95 TTFT measured with real support prompt lengths rather than average TTFT, and if a long system prompt is attached every time, it gives extra points to the one that supports prompt caching. If they run the model on their own servers, they adjust the number of concurrent requests in the inference engine while checking that TTFT does not exceed the target.