P²-MOPD: Accelerating Multi-Teacher On-policy Distillation with Parameter Priors

Problem

MOPD (Multi-Teacher On-policy Distillation) is used almost as a standard for merging domain-specific expert models into one. However, it requires student rollouts and teacher scoring at every step, which is costly, and recovery is uneven across domains. Training-free Task Arithmetic merging does not guarantee quality. In the authors' experiments, uniform merging recovered less than half of the gap between the base model and the teachers.

Method

Each module's weights are parameterized as W = B + Σ α_k·τ_k. B is the trainable student weight, τ_k are frozen expert task vectors, and α are trainable mixing coefficients. Training starts from the Task Arithmetic merge. The same gradient from the MOPD loss (where the domain's teacher scores the student-generated trajectory in reverse-KL form) updates two channels at once. One is the weight channel, which uses AdamW; the other is the coefficient channel, which uses the inner product of the gradient and τ. The authors interpret this structure as acting like a preconditioner that amplifies gradients in task-vector directions. Because the weights move freely, the model can also leave the space spanned by the task vectors.

Results

• On Qwen3-1.7B, average recovery at the end of training was 94.7%, higher than MOPD starting from the same TA merge (80.9%).
• The authors report that at both the 1.7B and 4B scales, it exceeded MOPD's end-of-training average recovery in half the steps, with lower wall-clock cost.
• Recovery in the worst-performing domain was below 50% with MOPD but rose to about 90% with P²-MOPD.
• A variant that trains only the coefficients and stays within the task-vector space performed much worse. The authors report that both the expert prior and the freedom to leave that space are necessary.

Limitations

Experiments are limited to two Qwen3 scales (1.7B and 4B) and four domains: medicine, law, finance and instruction following. All experts must derive from the same base model for task vectors to be defined. Only uniform values were used as initial merge coefficients. Frozen expert task vectors must be kept in memory during training, which may add memory overhead. The provided text is cut off in the experimental setup section, so only some detailed figures could be checked.

Why it matters

Training domain experts separately with reinforcement learning and then merging them into one general-purpose model is becoming more common in recent LLM development. This work combines distillation and model merging into a single training framework. It shows that faster, more balanced integration is possible with the same compute budget.

Key terms

On-policy Distillation

A distillation (knowledge distillation) method in which the teacher gives token-level feedback on responses (trajectories) that the student generates itself. It provides denser rewards than reinforcement learning.

MOPD

Short for Multi-Teacher On-policy Distillation. It guides the student with the teacher that matches the prompt's domain and is used to integrate multiple expert models into one.

Task Arithmetic / Task Vector

A task vector is the expert weights minus the base model weights. Task Arithmetic is a method that merges models without training by adding a weighted sum of these vectors to the base model.

Parameter prior

This means the expert weights are used not just once as initial values but as prior information that guides the update direction throughout training.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Hugging Face

Authors: Sinuo Wang, Jiacheng Lu, Zeyu Zhang, Maojia Song, Rui Sun, Beidi Luan, Yutong Xie, Jing Li, et al. · Posted 2026-11-02 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.