Paper review · November 14, 2026
Easing the learning–forgetting trade-off by raising the effective rank of LoRA adapters
Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off
Adding an SVD-free Rényi-2 regularizer that increases the effective rank of LoRA updates to the loss reduced forgetting in mixed-domain fine-tuning while maintaining learning performance.
Problem
LoRA performs worse than full fine-tuning and forgets more on heterogeneous data or in continual learning settings. The authors point to one cause: the update does not actually use the full configured rank r. Singular value energy is concentrated in a few directions. This tendency was stronger at larger r.
Method
The authors measured the capacity actually used by the LoRA update ΔW with the Roy–Vetterli effective rank. They then added to the language modeling loss a term that maximizes a lower bound on this value, the Rényi-2 effective rank erank₂ = ‖ΔW‖_F⁴ / Tr((ΔWᵀΔW)²) (in practice they use −λ·log erank₂). This term can be computed directly from the low-rank factors A and B without SVD, and it also applies to variants such as PiSSA and LoRA-XS. As an alternative, they also proposed an NB-LoRA path that caps the operator norm and increases the Frobenius norm.
Results
- The authors reported that in GSM8K, Hausa and mixed-domain rank sweeps, the erank/r ratio of standard LoRA was far below 1 and fell further as r increased. For Qwen2.5-0.5B at r=896, standard LoRA had an erank of 361.
- With regularization at λ=0.01, average Wiki perplexity (the forgetting metric) was lower than unregularized LoRA for Qwen2.5-0.5B, Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct, in every setting with rank 16 or above.
- Average training perplexity improved or rose by no more than 0.02 in most compared combinations. The one exception was r=16 on Qwen2.5-3B-Instruct.
- PiSSA and LoRA-XS had the highest Wiki perplexity (the most forgetting). VeRA, SVF and TinyLoRA reduced forgetting by learning less of the mixed data. The authors reported that adding the regularizer to PiSSA and LoRA-XS can improve both learning and retention.
- Regularized LoRA and standard LoRA had a similar share of singular values in directions that do not correspond to the pretrained basis (intruder directions) (0.97 vs 0.96). Even so, the regularized version forgot much less.
Limitations
The experiments cover 0.5B–7B models, the MLP down_proj layer, and a budget of about 1 million tokens. Some configurations (dense FT, λ=1, etc.) are single runs. The authors themselves state that the balance between learning and retention depends on settings and schedules. The NB-LoRA path showed unstable optimization in experiments. Forgetting was measured only through perplexity as a proxy metric.
Why it matters
The work tests the common belief that raising rank increases capacity, and shows that effective rank can be used as a training objective. Because it only requires adding one regularization term to existing LoRA training code, it is easy to apply in practice right away.
Key terms
- LoRA
- A parameter-efficient fine-tuning method that freezes the pretrained weights and trains only an update expressed as the product of two low-rank matrices.
- Effective rank
- A continuous rank value computed as the entropy of the normalized singular value distribution. It shows how evenly an update is spread across multiple directions.
- Rényi-2 effective rank
- A lower bound on effective rank, defined as the Rényi-2 entropy of the distribution of squared singular values. It can be computed differentiably from the Frobenius norm and trace, without SVD.
Original
Authors: Tomohiro Sawada, Kartik Goyal · Posted 2026-11-12 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.