Definition
QLoRA, or Quantized LoRA, is a parameter-efficient fine-tuning method that trains LoRA adapters while keeping the pretrained base model frozen and its weights stored in four-bit form. The base weights are dequantized to a separate compute dtype when needed for matrix multiplication. Training updates the adapters, not the base-model weights.
Storing the base weights in four bits reduces accelerator memory use compared with keeping the base model at higher precision. That can make larger models practical to fine-tune on smaller hardware.
Simple example
A team loads a compatible base language model with four-bit weights and adds trainable LoRA adapters to selected layers. Training updates only the adapter parameters. Afterward, the team stores the adapter separately and evaluates it with the same base-model and quantization configuration.
The quantized base weights remain unchanged throughout adapter training.
Why it matters
Base-model weights can take up a large share of fine-tuning memory. Quantizing them and training only the adapters avoids keeping the base weights at higher precision or allocating gradients and optimizer state for them. One frozen base model can then be reused with different adapters.
Training still needs memory for activations, adapters, gradients, optimizer state, and runtime workspaces. Data quality and evaluation matter as much as they do with other fine-tuning methods.
One important nuance
QLoRA uses quantization during adapter training, rather than only for inference. LoRA itself does not require a quantized base model. Implementations can differ in their four-bit formats, quantization schemes, supported layers, and deployment paths. A model trained with QLoRA will not necessarily serve faster, and deployment need not use the same quantization configuration.