LoRA: Fine-tuning a huge model by changing only small pieces

Why it was needed: the weight of full fine-tuning

LLMs (large language models) are usually pretrained on vast amounts of general data and then trained again for specific tasks such as summarization, question answering or SQL generation. This second step is fine-tuning. The simplest approach is full fine-tuning, which updates all of the model's parameters (the numbers the model sets through training).

The problem is size. Using GPT-3 175B as an example, the LoRA paper paper points out that building and deploying a separate 175-billion-parameter model for every task is prohibitively expensive. Making just three versions, for customer support, legal documents and code review, means three models the size of the original. Training also requires loading gradients and optimizer states (auxiliary values the training algorithm keeps for each parameter) for every parameter into GPU memory.

Methods that update only some parameters or attach small external modules already existed. But according to the same paper, these methods often made the model deeper and slowed inference, reduced the input length the model could use, or failed to match full fine-tuning. One had to give up either efficiency or quality.

Core idea: leave the original alone and just add sticky notes

Suppose you want to adapt a thick textbook to your own work. Instead of reprinting the whole book, you can leave the original as it is and add sticky notes where needed. When your work changes, you just swap the bundle of sticky notes. This is what LoRA (Low-Rank Adaptation) does.

The method works because of the assumption that "the change is simple." Building on earlier research, the LoRA paper paper assumed that the change in weights when a model adapts to a new task has a low "intrinsic rank." Put simply, rank is a number indicating how many independent directions a matrix actually moves in. If the change can be explained by just a few directions, there is no need to train the entire huge matrix; you only need to find those directions.

The results were large. Compared with full fine-tuning of GPT-3 175B with Adam, the same paper reported a 10,000-fold reduction in the number of trainable parameters and a 3-fold reduction in GPU memory requirements. Trainable parameters can shrink to about 0.01% of the original, and quality on RoBERTa, DeBERTa, GPT-2 and GPT-3 was reported to be on par with or better than full fine-tuning.

How it works: two small matrices next to frozen weights

Each layer of a transformer has large weight matrices that transform the input (for example, the query, key and value matrices in attention). LoRA keeps this original weight W frozen. It then attaches two small matrices alongside it. Matrix A compresses the input into a very narrow space (width r), and matrix B expands it back to the original size. The product of A and B stands in for the "change."

A single computation flows like this. 1) When input arrives, it passes through the frozen W to produce the original output. 2) The same input passes through A, becoming narrower, then through B, becoming wider again. 3) The two results are added to form the final output. 4) During training, only A and B are updated and W is left untouched. As a result, there is no need to keep gradients or optimizer states for most of the parameters.

How small can the width r be? The LoRA paper paper reported that for GPT-3 175B, even when the original dimension was 12,288, an r of 1 or 2 was sufficient. At deployment, B×A can be added to W in advance and merged into a single matrix. The computation path at inference then becomes identical to the original model, so the added modules cause no slowdown. If you keep A and B separate without merging, you can also use a single base model and swap in task-specific LoRAs.

How it developed: from LoRA to QLoRA

LoRA 2021 paper established the framework of "freeze the base model and train only small low-rank matrices." It greatly reduced trainable parameters and storage, made it possible to swap task-specific modules, and differed from earlier adapter methods in adding no inference latency. However, even during training, the frozen base model itself still had to sit in GPU memory at its original precision, such as 16-bit.

QLoRA 2023 paper reduced the weight of that very base model. According to the paper, fine-tuning LLaMA 65B in regular 16-bit requires more than 780GB of GPU memory. QLoRA quantizes the frozen base model to 4 bits (quantization is a technique that reduces size by representing numbers with fewer bits), attaches LoRA on top, and backpropagates gradients through the quantized weights into the LoRA part. As a result, it reported lowering the memory needed to fine-tune a 65B model to under 48GB while matching the performance of 16-bit full fine-tuning.

To do this, it added three mechanisms: NF4 (4-bit NormalFloat), a 4-bit data type suited to normally distributed weights; double quantization, which also quantizes the auxiliary constants used for quantization and saves about 0.37 bits per parameter on average (about 3GB for a 65B model); and paged optimizers, which offload memory spikes that occur when processing long inputs to NVIDIA unified memory. It also showed that applying LoRA to all layers is needed to largely avoid the accuracy loss seen in earlier work.

Using the resources saved this way, the QLoRA paper trained more than 1,000 models and also offered observations about data. It found that OASST1, with 9,000 samples, gave better chatbot performance than FLAN v2 with 450,000 samples (a subsample), and concluded that data quality matters far more than quantity. Guanaco, a model built this way, reportedly reached 99.3% of ChatGPT's performance on the Vicuna benchmark after 24 hours of training on a single GPU.

Where things are heading

First, theoretical work trying to explain "why LoRA works." The finite-rank bottleneck study paper notes that although LoRA has become the standard for parameter-efficient fine-tuning, the theoretical explanation for why a low rank is enough is still incomplete. The study focused on the fact that when the base model is frozen, the loss function sees each weight matrix only through a finite-rank channel formed by the frozen activations. It therefore showed that the required rank is determined not by the amount of data but by the rank of this channel, the "bottleneck rank."

Second, attempts to close the performance gap with full fine-tuning. DT-LoRA paper starts from the observation that LoRA often underperforms full fine-tuning. Its analysis found that the factorized update moves only within the space defined by the current low-rank factors, while gradient components outside that space account for most of the gradient energy. It therefore splits a fixed rank budget into a stable branch and a dynamic branch to capture the main directions outside that space. It reported consistent improvements on vision and language tasks without increasing trainable parameters or inference cost relative to LoRA.

Third, practical systems that "swap LoRAs." A medical image interpretation study paper built a system that attaches task-specific LoRA adapters to a single shared base model, Qwen3.5-9B frozen in 4 bits, and selects an adapter based on the incoming question. It covered six task groups and seven image types, with adapter rank and prompts tuned separately for each task. The final container was 30.1GB and took 3.1 seconds per case on average. The authors noted that the effect varied by task and interpreted the results as supporting per-task configuration. However, these results were measured on a competition evaluation set and do not indicate performance in real clinical settings.

Limitations and open issues

The performance gap does not always disappear. The LoRA paper paper reported quality on par with or better than full fine-tuning across several models, but DT-LoRA paper takes as its starting point that LoRA often underperforms full fine-tuning. Because the update is confined to a low-rank space, it may lose out when the change a task requires is not simple. Choosing the rank r and which layers to attach it to still largely has to be settled by experiment.

The explanation of "why it works" is also still in progress. As the finite-rank bottleneck study paper points out, most existing analyses have relied on linearly approximating the network near the pretrained weights. That a low rank is enough was at first closer to an empirical observation, and theoretical support is only now building up.

Evaluation issues follow as well. The QLoRA paper paper said GPT-4 evaluation broadly agrees with human evaluation on rankings and can be a cheap alternative, but also noted there are cases of strong disagreement and that chatbot benchmarks at the time were not reliable for accurately measuring performance. A high MMLU score also did not mean a high Vicuna chatbot score. Now that much more fine-tuning can be done at low cost, evaluation that confirms what actually improved has become more important.

Further study

The starting point is the original LoRA paper paper. Reading just Section 1 (introduction) and Section 2 (problem statement) gives you the storage and deployment burden of full fine-tuning and the idea of "expressing the change with a small number of parameters." Section 3, "Aren't Existing Solutions Good Enough?", compares the inference latency problem of adapter methods.

Next is QLoRA paper. If you want to fine-tune a large model on a single GPU in practice, it helps to know where NF4, double quantization and paged optimizers each save memory. Its analysis of data quality and evaluation methods is also worth reading.

After that, to follow recent directions, pick from three papers. If you are interested in theory, start with the finite-rank bottleneck study paper; if you are interested in variants that improve performance, DT-LoRA paper; and if you are interested in system design for running multiple tasks on one base model, the medical imaging adapter study paper.