Self-Distillation Fine-Tuning (SDFT): Learning New Skills Without Forgetting

Why it was needed: fine-tuning causes forgetting

The basic form of fine-tuning (further training an already-trained model on data for a specific task) is SFT (supervised fine-tuning). The model is given pairs of questions and reference answers and trained to reproduce each answer token by token. It is simple and effective, so it is part of almost all post-training.

The problem is that the reference answers were not written by the model itself. Answers written by people or other models differ in word choice, length and tone from this model's original distribution (what it considers plausible, and how plausible). Forcing the model to close this gap moves its internals substantially, and abilities it used to handle well, such as instruction following, safe responses and skills in other domains, break down. This is called catastrophic forgetting. The 'SDFT 2024' study paper identified the distribution gap between the task dataset and the LLM as the main cause of this problem.

The 'Continual Learning via Self-Distillation' study paper framed the same problem in terms of training methods. On-policy reinforcement learning, where the model learns from rollouts it generates itself, causes little forgetting but requires an explicit reward function. Such reward functions are often unavailable. So learning from demonstrations (reference answers) means using SFT, and SFT is inherently off-policy (learning from data produced by something other than the model). SDFT was introduced to fill the gap between these two approaches.

Core idea: don't copy the answer, restate it in your own words

Here is an analogy. SFT is like telling a student to copy a model answer without changing a single character. The student learns the content, but their usual handwriting and way of speaking get thrown off too. SDFT shows the student the model answer, says 'use this as a reference and rewrite it in your own words', and has them practice with what they wrote. The content they need to learn gets in, but the sentences are their own, so their original ability is damaged less.

Distillation (knowledge distillation) originally refers to having a small student model imitate the output distribution of a large teacher model. Self-distillation means the teacher and the student are the same model. The difference is that only the teacher side has extra information, for example by having seen the reference answer beforehand or by having a demonstration in its prompt. The teacher's output therefore becomes 'a distribution that carries the answer information but is what this model would naturally produce'.

The trade-off in this design is clear. The signal may be less sharp than using the external answer directly. In exchange, the training signal stays inside the model's distribution, which improves the balance between learning new things and keeping existing capabilities. There are two implementations under the same name SDFT, summarized in the comparison table below.

How it works: two kinds of SDFT

The first is the data-rewriting approach proposed in 'SDFT 2024' paper. (1) The model is given a question from the task data together with the original answer. (2) The model uses that answer as a reference and rewrites the answer in its own words. (3) Standard SFT is run on this distilled dataset instead of the original answers. The key is that the training data is aligned with the model's original distribution. In experiments with Llama-2-chat, the authors reported that forgetting was reduced while downstream task performance was comparable to or higher than standard fine-tuning, and that the method also showed promise for preserving helpfulness and safety alignment.

The second is the on-policy approach from the 'Continual Learning via Self-Distillation' study paper. (1) The student sees only the question, without a demonstration, and generates its own answer. This is the on-policy data. (2) The same model with the demonstration added to its prompt serves as the teacher. Thanks to in-context learning, the teacher makes better judgments than the student. (3) For each token of the student's answer, the probability distribution produced by the teacher is used as the training target, and the student is updated to reduce the difference between the two distributions. In effect, it obtains an on-policy learning signal from demonstrations alone, without a reward function.

Why the second approach reduces forgetting can be explained by the fact that the model 'learns along the path it actually takes'. SFT forces the model to follow sentence paths it would not take. On-policy SDFT, by contrast, makes only small corrections at each token along the trajectory the model generated itself. The 'Test-Time Self-Distillation' study paper interpreted this structure as the pointwise mutual information that arises from conditioning on the demonstration acting as an implicit reward. It looks like reinforcement learning, but no separate reward model needs to be built.

How it developed: recovery, information control, small models

The starting points are the two studies described above. 'SDFT 2024' paper mitigated forgetting with data that narrowed the distribution gap. 'Continual Learning via Self-Distillation' paper reported higher new-task accuracy than SFT on skill learning and knowledge acquisition tasks, with substantially less forgetting. In experiments that learned several skills in sequence, a single model accumulated skills without performance degradation. The two studies share a name but work differently.

The 'Self-Distillation as Performance Recovery' study paper used SDFT not only against forgetting but as a recovery tool to restore performance lost to quantization or pruning. The study argues that an LLM's generative ability depends on the high-dimensional structure (manifold) formed by its hidden layers, and used CKA (Centered Kernel Alignment) to measure how well the activation trajectories of the student and teacher match. Performance recovery turned out to be strongly correlated with the degree of alignment of this structure. This adds an internal-representation explanation to the question of why SDFT works.

iSDFT paper pointed out a weakness of on-policy SDFT. The existing method always pulls the student all the way toward the teacher that has seen the full demonstration, so there is no way to control how much demonstration information is passed at each token. iSDFT treats the teacher as 'a source that provides information up to a set budget' and, at each token, targets the distribution closest to the current student that satisfies the information constraint. To prevent cumulative drift, it also anchors to the frozen original model. Across four LLMs and two specialized tasks, it outperformed standard SDFT in 7 of 8 settings and matched it in the remaining 1. In 73% of evaluations it stayed within 0.5 points of the original model, compared with 52% for the strongest baseline.

SynthSelf paper addressed the fact that SDFT fails on small models. SDFT requires strong in-context learning and instruction-following abilities, which small models lack. The study showed that poor acquisition and forgetting are linearly correlated with the distribution shift after training, and assigned the rewriting work to a 'slightly larger' model from the same family. It combined this with off-policy supervision and rejection sampling (discarding low-quality generations). On 4 acquisition datasets and 14 retention benchmarks, it outperformed 6 methods at all 3 small model sizes.

Where it is heading now

The first direction is extending it to settings without demonstrations. SD-Zero paper has a single model play two roles. A generator first produces an answer, and a reviser looks at that answer and a correct/incorrect reward to produce a better one. The reviser's token distribution is then distilled into the generator through on-policy self-distillation. It is an attempt to turn the sparse binary reward of RLVR (reinforcement learning with verifiable rewards) into a dense token-level signal. On Qwen3-4B-Instruct and Olmo-3-7B-Instruct, the authors reported improvements of at least 10% over the original models and better results than RFT, GRPO and SDFT under the same question set and training sample budget.

The second is applying the same principle at inference time without training. 'Test-Time Self-Distillation' paper puts fixed phrases into the context instead of demonstrations, describing a 'reasoning very well' scenario and a 'reasoning very poorly' scenario. It then uses the log-probability ratio of a candidate answer under the two conditions as the reward. The distribution reweighted by this reward cannot be factored token by token, so it is approximated with beam search. The authors stated that it outperformed standard sampling, low-temperature sampling and standard beam search on MATH500, HumanEval and GPQA.

The third is building it into real domain post-training. A study on post-training for financial reasoning paper used self-distillation SFT and model merging, alongside SFT and reinforcement learning, as 'mechanisms for not losing existing financial capabilities'. On FINESSE-Bench, standard SFT lowered accuracy by 3.2–4.0 percentage points, while self-distillation SFT raised it by 1.0–2.8 points over the starting model. Equal-weight merging recovered 3.0 points over the SFT model, ending 0.9 points above the original model. On the agent side, ACLArena paper systematically compared how much multi-teacher on-policy distillation, self-distillation fine-tuning and model merging restore earlier capabilities in multi-stage post-training.

Limitations and open issues: when to use it and when to avoid it

The biggest prerequisite is that the model must make good use of demonstrations in context. As SynthSelf paper showed, on-policy SDFT can fail on small models. For small on-device models, consider the workaround of handing the rewriting to a slightly larger model from the same family. Also, because the teacher is ultimately the model itself, when injecting knowledge the model does not have at all, how accurate the signal can be is bounded by the teacher's level.

The second issue is 'how closely to follow the teacher'. As iSDFT paper pointed out, passing all demonstration information unconditionally leaves no knob for balancing specialized performance against general capabilities. Passing too little, on the other hand, means the new task is learned less. Even under the same name, the data-rewriting type paper and the on-policy type paper have different cost structures. The rewriting type only needs one round of generation followed by standard SFT. The on-policy type must keep generating and computing teacher probabilities during training, so training costs more.

The practical criteria are as follows. It is worth considering when adapting a general-purpose assistant to a specific task while preserving safe responses or instruction following, when adding capabilities in sequence over several stages, or when you want to restore performance lost after quantization. Conversely, if the goal is to completely change the model's tone or format itself, SDFT's tendency to preserve the original distribution may get in the way. A common mistake is judging only by the new-task score. The value of this technique lies in retained performance, so changes on general benchmarks relative to the original model should be measured as well. As in the finance study paper, it is also good to compare it against other options such as model merging.

When applying it to a domain, verification mechanisms should be put in place too. In fields where knowledge consistency matters, such as medicine, Evontree paper first used ontology rules to find inconsistencies in the model's knowledge and then filled the gaps with self-distillation fine-tuning. It reported accuracy gains of up to 3.7% on medical QA. Signals the model generates itself can also carry its own errors. So a filtering step based on external criteria such as rules or verifiers is effectively essential.

Further reading

Start by reading the two starting points side by side. 'SDFT 2024' paper presents the problem framing of the distribution gap and the simplest solution, data rewriting. Its code is public, which makes it easy to follow along. Reading 'Continual Learning via Self-Distillation' paper next makes it clear why the same problem is tackled with on-policy learning.

After that, choose by interest. If you are curious about the underlying principle, read the 'performance recovery' study paper, which explains it through internal representation alignment. If you need a controllable variant, iSDFT paper fits; if your target is small models, SynthSelf paper. If you are curious about the connection to reinforcement learning, SD-Zero paper and 'Test-Time Self-Distillation' paper are good choices.

For practical applications, the financial post-training pipeline paper and the comparison of continual learning for agents (ACLArena, paper) compare SFT, model merging and self-distillation under the same conditions, which helps in setting selection criteria.