What Learning Rate Should You Use for QLoRA Fine-Tuning

TL;DR

Learning rate is the hyperparameter with the most impact on fine-tuning outcomes and the most potential to silently ruin a training run. Too high and you get loss spikes or divergence. Too low and training stalls or takes far longer than it should to converge. This post covers the recommended starting values for QLoRA fine-tuning at different model sizes, what the research literature converges on, how to read your loss curve to know if your rate is wrong, and when and how to adjust.

If you are copying configs from tutorials and wondering why your results are different, learning rate is usually the first place to look.

It is the single hyperparameter that has the most influence over whether fine-tuning converges cleanly, converges slowly, or does not converge at all. And the right value is not the same for every job. It depends on your model size, your LoRA rank, your dataset, and your batch size.

Here is how to set it correctly and how to know when to change it.

Why Learning Rate Is Different for QLoRA vs Full Fine-Tuning

In full fine-tuning, you are updating all parameters of the model. Learning rates for full fine-tuning are typically in the range of 1e-5 to 5e-5. Small adjustments to billions of parameters.

In LoRA and QLoRA, you are only updating the adapter parameters, a much smaller set. The adapter weights start at or near zero and need to move significantly from their initialization to encode the behavior you want. This means LoRA adapters can tolerate and often benefit from higher learning rates than full fine-tuning.

For QLoRA specifically, the quantized base model is frozen. The adapters are the only thing learning. The effective learning signal is the same but it flows into a much smaller set of parameters. This is why QLoRA learning rates are typically 10 to 100 times higher than full fine-tuning rates.

The Recommended Starting Values

These are the ranges the fine-tuning community has converged on through practice and research as of 2026. They are starting points, not guarantees.

7B to 13B models with QLoRA: 2e-4 (0.0002) is the most commonly used starting point. This is what the original QLoRA paper used, what most community configs default to, and what works reliably across a wide range of tasks. Start here unless you have a specific reason to do otherwise.

1B to 3B models with QLoRA: Same range applies. Smaller models are not inherently more sensitive to learning rate. 2e-4 works as a starting point here too, as confirmed in the real training run on Llama 3.2 1B covered in this blog's previous post.

30B to 70B models with QLoRA: Slightly lower. 1e-4 to 1.5e-4 is a more conservative starting point for larger models. The adapter parameter count is similar but the base model representations are richer and more sensitive to disruption from large updates.

Higher lora_r values: A higher LoRA rank means more trainable parameters. More parameters absorbing the same gradient signal means each parameter moves less per step. You can compensate by slightly increasing the learning rate, but the relationship is not linear. If you are using lora_r: 64 instead of lora_r: 16, try 3e-4 rather than jumping to 8e-4.

The Learning Rate and Batch Size Relationship

Learning rate does not exist in isolation. It interacts with your effective batch size.

The linear scaling rule says: if you double your batch size, you can roughly double your learning rate while maintaining similar training dynamics. This is because a larger batch gives you a more accurate gradient estimate, which means each update is more reliable and you can afford to move faster.

In practice for fine-tuning, this relationship is approximate rather than exact. But the direction is right. If you increase gradient_accumulation_steps from 4 to 8, effectively doubling your batch size, you can try increasing your learning rate slightly. If you reduce batch size due to memory constraints, consider reducing learning rate correspondingly.

The config from the real training run on this blog used:

micro_batch_size: 2

gradient_accumulation_steps: 4

learning_rate: 0.0002

Effective batch size of 8 with 2e-4. That is a consistent combination. If you copy this config and change only the batch size without adjusting the learning rate, your results may differ from what you expect.

How to Read Your Loss Curve for Learning Rate Problems

Your loss curve is the fastest feedback signal on whether your learning rate is right.

Learning rate too high:

The loss drops quickly at first and then becomes erratic. Spikes up, recovers, spikes again. Or it drops fast and then starts climbing. The grad_norm in your training logs will also be high and variable, often above 10.

The fix: reduce your learning rate by half and rerun. If you were at 2e-4, try 1e-4.

Learning rate too low:

The loss drops very slowly and steadily across the entire training run without clearly converging. At the end of training it is still meaningfully decreasing, suggesting the model has not reached a good minimum. The grad_norm will be small and consistent, often well below 1.0.

The fix: increase your learning rate by 2x to 5x. If you were at 2e-5, try 1e-4.

Learning rate approximately right:

Loss drops steadily in the early epochs, then flattens and stabilizes as the model converges. The learning rate schedule decays it toward zero by the end of training, and the final loss is stable rather than still falling. Grad norm is moderate, typically between 0.5 and 5.0, and decreases as training progresses.

This is what you want. The real training run in the previous post showed exactly this pattern, with loss stabilizing around 0.8 to 0.93 in the final steps and learning rate decayed to near zero by epoch 3.

The Learning Rate Schedule Matters Too

Learning rate is not a fixed value during training. The scheduler controls how it changes over time.

For QLoRA fine-tuning, cosine decay is the most widely recommended schedule. It starts at your target learning rate, decreases following a cosine curve, and reaches near zero by the end of training. It gives the model more time at higher learning rates early, when there is more to learn, and brings it down gradually as convergence approaches.

learning_rate: 0.0002

lr_scheduler: cosine

warmup_ratio: 0.03

This is the combination that appears in most well-performing QLoRA configs. The warmup_ratio of 0.03 ramps the learning rate from zero to 2e-4 over the first 3 percent of training steps, protecting early training from large updates before the optimizer stabilizes.

Avoid constant learning rate for multi-epoch runs. Without decay, the model keeps making full-size updates in the final steps when it should be making fine adjustments. This often produces worse final models than cosine decay at the same peak rate.

A Practical Approach to Learning Rate Tuning

If you are unsure where to start, follow this sequence.

Run one short diagnostic job at 2e-4 for a fraction of your planned total steps, maybe 10 to 20 percent. Look at the loss curve. Is it dropping smoothly? Is it spiking? Is it barely moving?

If smooth: continue with the full run at 2e-4.

If spiking: halve the learning rate. Run another diagnostic at 1e-4.

If barely moving: double the learning rate. Run another diagnostic at 4e-4.

A diagnostic run costs very little compute. The information it gives you is worth more than running a full job at a rate that turns out to be wrong.

For most 7B QLoRA jobs on tasks like instruction following, summarization, or classification, 2e-4 with cosine decay and 3 percent warmup is the right starting point and you will not need to adjust it. The range where things start going wrong is below 5e-5 on the low end and above 5e-4 on the high end.

The Config Line That Matters

For a standard 7B QLoRA job, this is what you want:

learning_rate: 0.0002

lr_scheduler: cosine

warmup_ratio: 0.03

Change only one of these at a time when troubleshooting. If your loss is spiking, adjust learning_rate first. If your loss is barely moving, adjust learning_rate first. The scheduler and warmup are usually not the problem.