What Is a LoRA Adapter and How Does It Work

TL;DR

LoRA, which stands for Low-Rank Adaptation, is the technique that made practical LLM fine-tuning accessible to engineers without access to massive GPU clusters. Instead of updating all the weights in a model during training, LoRA freezes the original model and adds a small set of new weight matrices that are trained alongside it. The result is a small, portable adapter file that encodes the fine-tuned behavior separately from the base model. This post explains how LoRA works mechanically, what the rank and alpha settings actually control, and what you get at the end of a LoRA training run.

If you have spent any time around fine-tuning, you have heard the word LoRA more times than you can count.

It shows up in configs. It shows up in research papers. It shows up in every tutorial and every benchmark comparison. And most explanations either go too deep into the math or stay too shallow to actually be useful.

Here is the version that tells you what you actually need to know.

The Problem LoRA Was Designed to Solve

To understand LoRA you need to understand what it was solving.

Full fine-tuning updates every weight in the model. For a 7B parameter model, that means 7 billion numbers being updated on every training step. To do that you need to store the model weights, the gradients for each weight, and the optimizer states for each weight all in GPU memory simultaneously. On a 7B model that adds up to 60 to 80 GB of GPU memory. A single A100 has 80 GB. Most accessible cloud GPUs have 24 GB.

Fine-tuning was expensive, slow, and inaccessible to most teams. The LoRA paper, published by Microsoft researchers in 2021, proposed a way to dramatically reduce that cost without meaningfully hurting results.

What LoRA Actually Does

Here is the core idea.

In a neural network, the weight matrices that do the most important work, the attention projection matrices in a transformer, are large. A typical weight matrix in a 7B model might be 4096 by 4096. That is about 16 million numbers per matrix.

LoRA observes that the changes needed to fine-tune a model for a specific task can often be represented as a low-rank update. In other words, you do not need to change all 16 million numbers in a matrix. You can capture the important changes with a much smaller pair of matrices that multiply together to approximate the full update.

Concretely, instead of updating the full weight matrix W directly, LoRA freezes W and adds two small matrices A and B alongside it. During training, only A and B are updated. Their product A × B represents the adaptation. The effective weight used during the forward pass is W plus the result of A × B.

If W is 4096 by 4096 (about 16 million parameters), and you set the LoRA rank to 16, then A is 4096 by 16 and B is 16 by 4096. That is about 130,000 parameters total. Instead of updating 16 million numbers per matrix, you are updating 130,000.

Across a full 7B model, this reduces the number of trainable parameters from billions to millions. The memory savings for gradients and optimizer states are proportional. That is what makes LoRA trainable on a 24 GB GPU.

What lora_r and lora_alpha Actually Control

These are the two LoRA settings you will see in every config. Here is what they do.

lora_r is the rank. It controls the size of the A and B matrices. A rank of 16 means A is [model_dim × 16] and B is [16 × model_dim]. A rank of 64 means A is [model_dim × 64] and B is [64 × model_dim].

Higher rank means more trainable parameters. More parameters means the adapter has more capacity to represent complex behavioral changes. But it also means more memory and more compute per step.

For most fine-tuning tasks, a rank of 8 to 32 is sufficient. Rank 16 is the most commonly used default. Rank 64 or higher is worth trying only if you have confirmed that a lower rank is not capturing what you need.

lora_alpha is a scaling factor applied to the adapter output. The effective update applied to the original weights is (lora_alpha / lora_r) times the product A × B.

Setting lora_alpha to twice lora_r is a widely used convention. So lora_r: 16 with lora_alpha: 32 gives a scaling factor of 2. This means the adapter contribution is amplified relative to a scaling factor of 1, which helps the adapters have more influence without requiring a higher learning rate.

You will see lora_alpha set to the same value as lora_r in some configs. That gives a scaling factor of 1. Both approaches work. The double alpha convention is more common because it tends to require less learning rate tuning.

What lora_target_modules Controls

LoRA does not have to be applied to every layer in the model. You choose which weight matrices to adapt.

In transformer models, the most commonly targeted modules are the attention projection matrices: q_proj, k_proj, v_proj, and o_proj. These are the weights that control how the model pays attention to different parts of the input sequence.

Many configs also include the MLP layers: gate_proj, up_proj, and down_proj. Including these gives the adapter more capacity to modify the model's factual and reasoning behavior, not just its attention patterns.

In Axolotl, setting lora_target_linear: true applies LoRA to all linear layers in the model, which includes both attention and MLP. This is the most comprehensive option and generally produces better results at the cost of more trainable parameters.

For a first run, lora_target_linear: true is a reasonable default. You can experiment with more selective targeting if you need to reduce memory pressure.

What You Get at the End of Training

When LoRA training completes, the output is an adapter file. Not a full model. Just the A and B matrices for each targeted layer, saved in safetensors or bin format.

This adapter file is small. For a rank 16 LoRA adapter on a 7B model, the adapter file is typically 20 to 50 MB. The full 7B base model is 13 to 14 GB in 16-bit precision.

To use the fine-tuned model at inference, you load the base model and apply the adapter on top of it. The adapter adds its contribution to the base model weights at each targeted layer. From the outside, it looks and behaves like a single model.

from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
model = PeftModel.from_pretrained(base_model, "./your-lora-adapter")

If you want a single merged model file rather than a base plus adapter, you can merge the adapter weights into the base model:

merged_model = model.merge_and_unload() merged_model.save_pretrained("./merged-model")

The merged model behaves identically to the base plus adapter but does not require loading two separate files.

Why LoRA Adapters Are Powerful Beyond Memory Savings

The memory efficiency is the headline. But LoRA has another advantage that is easy to overlook.

Because the adapter is separate from the base model, you can have multiple adapters trained for different tasks and swap between them at inference time without reloading the base model. A single deployed base model can serve many different fine-tuned behaviors by loading different adapters.

For teams serving multiple use cases, this is a meaningful cost reduction. One 7B base model loaded in memory. Adapters swapped in and out per request. No multiple full model deployments.

This is not possible with full fine-tuning, where each fine-tuned version is a separate full model.

The Short Version

LoRA freezes the original model, adds small trainable matrices at specific layers, and trains only those matrices. You get a small adapter file that encodes the behavior change. Load it on top of the base model and you have a fine-tuned model that required a fraction of the memory and compute of full fine-tuning.

That is why LoRA is everywhere. It made fine-tuning practical for the hardware most engineers actually have.