AI glossary · Basic · Efficiency & infra
Inference
Inference is the process of feeding new input into an AI model that has already finished training to get actual results (answers, images, classifications and so on). In other words, it is the stage of 'using' the model.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Foundations & applications
- Fields
- Efficiency & infra
- Added
- 2026-10-08
How it works
An AI model's life is broadly split into training and inference. In training, the model looks at large amounts of data and adjusts the numbers inside it (parameters); in inference, those numbers stay fixed and the model only computes on new input. For an LLM, the prompt is split into tokens and fed into the model, which predicts the single next token. It appends that token to the input and predicts the next token again, repeating until the whole answer is complete.
Why it matters
Training happens once on a large scale, but inference happens every time a user asks a question. So as a service grows, a substantial share of total cost and power goes to inference. User experience factors such as response speed, pricing and whether a model can run directly on a smartphone (on-device AI) also depend on inference efficiency. That is why techniques that make inference faster and cheaper, such as quantization, KV cache, speculative decoding and dedicated chips (NPUs, etc.), are actively researched.
Things to know
In Korean, the word '추론' is used for two meanings and is easy to confuse: inference, which means running a model to get results, and reasoning, which means thinking things through step by step. Terms like 'inference cost' or 'inference server' usually refer to inference, while 'reasoning model' or 'reasoning ability' refer to reasoning. Also, because the model does not learn anything new during inference, information you give it in a conversation is only used within that conversation's context and is not stored in the model itself.
Example
The moment you type a question into a chatbot and the answer appears character by character is inference in action. AI service companies run this process on GPU servers and charge API fees by the number of input and output tokens, because the computation needed for inference grows in proportion to the number of tokens. A smartphone photo app that recognizes objects in photos without an internet connection is also an example of performing inference on the device.