With more pretraining data, the fine-tuning advantage between Muon and AdamW flips

Problem

The Muon optimizer lowers pretraining loss faster than AdamW. In practice, however, what matters is how well a model learns the target task after fine-tuning while forgetting as little existing knowledge as possible (the learning-forgetting trade-off). It was not known whether Muon's advantage also holds by this measure.

Method

OLMo-family models from 30M to 600M parameters were pretrained on 150 million to 154 billion DCLM tokens, once with AdamW and once with Muon (about 600 runs). Each model was then fine-tuned on GSM8K, StackMathQA, Alpaca and MusicPile with different optimizers and learning rates (about 29,000 runs). The two optimizers were compared on the Pareto frontier of fine-tuning loss (learning) and pretraining loss (forgetting). To find the cause, the authors defined the Margin-to-Spread Ratio (MSR): how far the correct logit sits above the incorrect logits, divided by the spread of the incorrect logits. They also analyzed the one-step updates of GD and Muon in a linear associative memory theoretically.

Results

• When OLMo-60M was fine-tuned on StackMathQA, Muon's frontier dominated AdamW's after 2.4B tokens of pretraining. With more tokens than that, AdamW showed both lower fine-tuning loss and less forgetting. Muon had lower pretraining loss at every data scale.
• The token-to-parameter ratio at which the reversal happens (crossover TPR) got smaller as models got larger. On StackMathQA it was 320 for the 30M model and 10 for the 600M model.
• The authors reported that the same reversal appears even when the pretraining setting with the best frontier is chosen instead of the model with the lowest pretraining loss.
• In language models, MSR was larger for Muon when data was limited and larger for AdamW when data was plentiful. This matches the trend in the fine-tuning results.
• In the one-step analysis of a linear associative memory, the authors proved that the GD/Muon MSR ratio grows as the data budget increases and approaches 3π/8 (about 1.18) from below.

Limitations

The experiments are limited to models with 600M parameters or fewer. Learning and forgetting were measured by loss, not by downstream task accuracy. The theoretical analysis compares full-batch GD, not AdamW, with Muon using exact orthogonalization. It also analyzes a single update step starting from zero, so multi-step training was checked only in simulation. The crossover point depends on the fine-tuning dataset.

Why it matters

The paper shows that judging a pretraining optimizer by pretraining loss alone can give the wrong picture of performance after fine-tuning. With the recent trend toward token-heavy pretraining, and with larger models, Muon's advantage can disappear or reverse at the fine-tuning stage.

Key terms

Muon

An optimizer that updates weights in the direction of the orthogonalized gradient matrix (all singular values set to 1). It is known to lower LLM pretraining loss faster than AdamW.

Learning–forgetting trade-off

The trade-off between how well fine-tuning teaches a model a new task and how much pretrained knowledge the model loses. This paper compares it using the Pareto frontier of the two losses.

Margin-to-Spread Ratio(MSR)

The difference between the correct logit and the mean of the incorrect logits, divided by the standard deviation of the incorrect logits. The larger it is, the more likely the correct answer holds even when outputs fluctuate.

Token-Parameter Ratio(TPR)

The number of pretraining tokens per model parameter. It is used to compare pretraining data budgets relative to model size.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Catherine Li, Jingchu Gai, Andrej Risteski, Jacob Mitchell Springer, Aditi Raghunathan · Posted 2026-11-02 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.