← Key papers

Paper review · November 14, 2026

Easing the learning–forgetting trade-off by raising the effective rank of LoRA adapters

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

Language modelsEfficiency & infra Advanced cs.LG

Adding an SVD-free Rényi-2 regularizer that increases the effective rank of LoRA updates to the loss reduced forgetting in mixed-domain fine-tuning while maintaining learning performance.

Problem

LoRA performs worse than full fine-tuning and forgets more on heterogeneous data or in continual learning settings. The authors point to one cause: the update does not actually use the full configured rank r. Singular value energy is concentrated in a few directions. This tendency was stronger at larger r.

Method

The authors measured the capacity actually used by the LoRA update ΔW with the Roy–Vetterli effective rank. They then added to the language modeling loss a term that maximizes a lower bound on this value, the Rényi-2 effective rank erank₂ = ‖ΔW‖_F⁴ / Tr((ΔWᵀΔW)²) (in practice they use −λ·log erank₂). This term can be computed directly from the low-rank factors A and B without SVD, and it also applies to variants such as PiSSA and LoRA-XS. As an alternative, they also proposed an NB-LoRA path that caps the operator norm and increases the Frobenius norm.

Results

Limitations

The experiments cover 0.5B–7B models, the MLP down_proj layer, and a budget of about 1 million tokens. Some configurations (dense FT, λ=1, etc.) are single runs. The authors themselves state that the balance between learning and retention depends on settings and schedules. The NB-LoRA path showed unstable optimization in experiments. Forgetting was measured only through perplexity as a proxy metric.

Why it matters

The work tests the common belief that raising rank increases capacity, and shows that effective rank can be used as a training objective. Because it only requires adding one regularization term to existing LoRA training code, it is easy to apply in practice right away.

Key terms

LoRA
A parameter-efficient fine-tuning method that freezes the pretrained weights and trains only an update expressed as the product of two low-rank matrices.
Effective rank
A continuous rank value computed as the entropy of the normalized singular value distribution. It shows how evenly an update is spread across multiple directions.
Rényi-2 effective rank
A lower bound on effective rank, defined as the Rényi-2 entropy of the distribution of squared singular values. It can be computed differentiably from the Frobenius norm and trace, without SVD.

Original

Authors: Tomohiro Sawada, Kartik Goyal · Posted 2026-11-12 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.