AI Trend Research

← AI glossary

AI glossary · Advanced · Efficiency & infra

Distillation

Distillation (knowledge distillation) is a method of training a smaller, faster student model to reach similar performance by using the answers of a large, high-performing teacher model as examples.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-08
In plain words When an experienced senior shares not only how they solve a problem but also a sense of "this answer is likely, and that one is also somewhat plausible," a junior improves much faster than by studying only the textbook. However, because the junior copies the senior directly, they can also learn the senior's mistakes.

How it works

First, a large, accurate teacher model is prepared, and the same input is fed to both the teacher and the student model. The student is trained to reduce the difference between the two, so that it imitates the teacher's answers or the probability distribution over which answer is correct. Looking at the teacher's probabilities, which also capture "how plausible the other answers are," lets the student learn more than seeing only the single correct answer. For language models, it is also common to use responses or solutions generated by the teacher as training data. Variants such as on-policy distillation, in which the teacher gives feedback on answers the student generates itself, are also being studied.

From teacher to studentInput data → Teacher model; Input data → Student model; Teacher model → Teacher's answer; Student model → Student's answer; Teacher's answer → Reduce the difference(Example); Student's answer → Reduce the difference; Reduce the difference → Student model(Update)Input dataTeacher modelLarge and accurateTeacher's answerAnswers, probabilitydistributionStudent modelSmall and fastStudent's answerReduce thedifferenceExampleUpdate
From teacher to student The same input is fed to both models, and only the student is updated to reduce the difference so that its answers move closer to the teacher's.

Why it matters

Large models perform well but are expensive and slow to run, which makes them hard to use as is on smartphones or in real-time services. With distillation, a small model inherits much of a large model's capability and can be used more cheaply and quickly. That is why it is used as a key technique for building SLMs (small language models) and on-device AI. In diffusion models, it is also used to shorten a multi-step generation process into fewer steps.

Things to know

The student generally struggles to surpass the teacher and can also inherit the teacher's errors and biases. If the two models use different tokenizers (ways of splitting text into tokens), it is hard to compare their answers token by token, so extra handling is needed. Distilling the outputs of another company's model can raise disputes over terms-of-use violations, so care is needed. It is often used together with other techniques for making models smaller, such as quantization and pruning.

Example

The Paradee news covered on this site is a good example. It describes using the already released speech synthesis model Kokoro-82M as the teacher and distilling it into a small 8M-parameter TTS model that produces only one person's voice. Transferring a large model's capability to a small model like this makes it possible to run it directly on a device or lower service costs, so many companies use this method when building smaller versions of their own large models.

Where it came up on this site

Related terms

SLM On-device AI Quantization SFT Post-training Diffusion model