Self-Distillation Fine-Tuning

What it is

SDFT is a technique proposed in the paper 'Self-Distillation Enables Continual Learning', posted on arXiv in January 2026 and revised in August of the same year. According to a report in a Korean IT outlet, it was developed by researchers from MIT, Improbable AI Lab and ETH Zurich. The problem it targets is 'catastrophic forgetting', in which a model loses the ability to do what it originally did well while learning a new task. SDFT aims to replace SFT (supervised fine-tuning, which trains a model to copy correct examples as given) and let a model keep building new skills with only demonstration data. Learning by accumulating skills this way is called continual learning.

How it works

During training, one model plays two roles. The teacher role receives the question together with an expert demonstration example. Because models can pick up and follow examples given in the prompt (in-context learning), the model produces better answers when it has seen the example. The student role receives only the question, as in real use, and writes its own answer. Then, using the answer the student wrote, the model's parameters are adjusted to bring it closer to the teacher's prediction for that answer. Because the model learns from outputs it generated itself, this is on-policy learning.

Why it is drawing attention

Until now there were two approaches. Reinforcement learning has the model learn from data it generates itself, so forgetting is lower. But it needs a reward function that scores what a good answer is, and such a function is often hard to define. SFT needs only demonstrations, but it is off-policy, copying someone else's answers rather than ones the model produced, so forgetting is high. The researchers say SDFT combines the strengths of both by enabling on-policy learning from demonstrations alone. They report that on skill-learning and knowledge-acquisition tasks it achieved higher accuracy on new tasks than SFT with much less forgetting, and that performance did not regress when it learned multiple skills in sequence. For companies, there is hope that it could reduce the burden of managing many separately fine-tuned models or adapters, one per task.

Things to know (limitations)

As of October 2026, SDFT is a research-stage technique, and it is not yet clear whether it will be widely used in commercial services. According to the report, training takes considerably longer than SFT and uses about 2.5 times the compute. It also requires a sufficiently strong base model that is good at in-context learning for the teacher role to work properly. Some point out that because the model learns from its own rollouts, reproducing results requires careful version control and output logging. It also does not remove the need to re-verify that performance has not dropped. For these reasons, some analysts expect it to be tried first in internal developer tools or general-purpose assistants rather than in heavily regulated fields such as finance and healthcare.

Example

Suppose a company wants to teach its internal coding assistant how to use a newly introduced internal API. Developers have written model code examples (demonstrations), but there is no reward function to automatically score whether code is good. Training with SFT would teach the new API but risks degrading the general coding skills the assistant already had. With SDFT, the model that sees the model examples (teacher role) corrects the answers of the model that sees only the question (student role). This lets the assistant add knowledge of the new API while keeping its existing skills. In exchange, training costs more, and existing tasks still need to be re-evaluated after the update.

Related terms

Distillation SFT Fine-tuning Reinforcement learning Post-training Evals

References

• Self-Distillation Enables Continual Learning (arxiv.org)
• 'Catastrophic forgetting,' a chronic problem in LLM updates: SDFT attempts a breakthrough (itworld.co.kr)