Definition
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique. It keeps the pretrained model weights frozen and, for each selected weight matrix, trains two smaller matrices whose product defines an additive update to that weight matrix.
The adapter has far fewer trainable parameters than the base model. Teams can store it separately or merge the update into compatible model weights for deployment.
Simple example
A team wants a language model to produce incident summaries in its internal format. It adds LoRA adapters to selected transformer projections and trains only those parameters on reviewed examples. The adapter is much smaller than a complete model copy. The same base model can support adapters for other tasks.
The team should evaluate the target task and important general behavior, ideally against the unadapted base model.
Why it matters
LoRA reduces the number of trainable parameters and the optimizer state needed during training. This can lower training memory use and make it practical to maintain several task-specific adapters for one base model.
At deployment, a team can load an adapter alongside the base model or merge its update into the model weights. Either way, it needs to track which adapter and base-model versions are compatible.
One important nuance
LoRA is one parameter-efficient fine-tuning (PEFT) method, not the name for the whole family. Rank, target weight matrices, training data, and optimization settings affect what the adapter learns. Quantization is a separate technique. A quantized base model can reduce memory use during LoRA training, while serving may use a different weight representation. Compatibility depends on each setup.