AI glossary · Beginner · Language models
Token
A token is the smallest unit an AI language model uses when processing text. It may be a whole word, part of a word, or a few characters.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Basics
- Fields
- Language models
- Added
- 2026-10-07
How it works
When you give text to an AI, a tool called a 'tokenizer' first cuts it into tokens. Common words become a single token, while rare words are split into several tokens. Each token is converted into a number and fed into the model, which writes its answer by predicting the next token one at a time.
Why it matters
Many AI services charge based on the number of tokens used. The amount a model can read at once (the context window) is also set in tokens. So understanding tokens helps you estimate costs and how long a document can be processed.
Things to know
The same content can require a different number of tokens depending on the language. In general, languages such as Korean are often split into more tokens than English. Each model also has its own tokenizer, so the same text yields different token counts on different models.
Same meaning, different number of pieces
Example
AI service price lists often show separate prices for 'input tokens' and 'output tokens'. This site's news has also included a case that introduced a new API's price on a 'per million tokens' basis; tokens are used in this way as the basic unit for measuring AI usage.
Where it came up on this site
- briefing OpenAI launches 'Decisions API' based on decision-only model 'gpt-6-luna': $0.10 per million input tokens, no charge for output (community report) 2026-10-07
- Papers ALoDLM: a looped diffusion language model that adjusts recurrence depth per token 2026-10-07
- briefing Study: fixing only the response's opening tokens brings base model reasoning performance close to RL models 2026-10-06