CUDA Out of Memory When Fine-Tuning: What's Actually Happening and How to Fix It

TL;DR

CUDA out of memory is the most common error engineers hit when fine-tuning large language models. The error itself is clear. The cause is usually not. GPU memory fills up for several different reasons during training, and the fix depends entirely on which one you are hitting. This post explains what is actually consuming your GPU memory, how to diagnose which cause is behind your specific error, and the fixes for each one in order of what to try first.

You hit run. The job starts. And then somewhere in the first few steps you get this:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 23.69 GiB total capacity; 21.23 GiB already allocated; 1.06 GiB free; 21.89 GiB reserved in total by PyTorch)

The instinct is to cut the batch size to 1 and try again. Sometimes that works. A lot of the time it does not, because batch size is only one of several things filling your GPU memory. And if you are hitting the wrong fix for your actual problem, you will burn an hour cycling through guesses.

Here is the systematic version.

What Is Actually in Your GPU Memory During Training

Before you can fix the problem you need to know what is competing for space.

During a QLoRA or LoRA training run, your GPU is holding several things at once.

The model weights take the largest chunk. A 7B model in 4-bit precision takes roughly 4 to 5 GB. In 16-bit it takes 14 GB. In 32-bit it takes 28 GB. If you are not quantizing, this alone fills most of a 24 GB GPU before training even starts.

The activations are the intermediate values computed at each layer during the forward pass. These scale with your sequence length and batch size. Long sequences and large batches produce large activations.

The gradients track how much each parameter should change based on the loss. For LoRA and QLoRA only the adapter parameters have gradients, which is why these methods use so much less memory than full fine-tuning.

The optimizer states store momentum and variance information for each trainable parameter. For AdamW this is roughly 2x the size of your trainable parameters.

The KV cache can accumulate during training on long sequences and eat memory in ways that are not obvious from the error message.

All of these stack on top of each other. The OOM error fires when the total exceeds your GPU capacity. Which piece pushed you over tells you what to fix.

Step One: Check What You Are Actually Using

Before changing anything, run this and look at the numbers:

nvidia-smi

This shows you how much memory is currently allocated on your GPU. If you are already at 22 GB out of 24 GB before training even starts, the problem is model loading. If you start at 8 GB and spike to OOM during the first forward pass, the problem is activations or sequence length.

Knowing which phase the OOM hits tells you where to look.

The Most Common Causes and Their Fixes

You are not quantizing and your model is too large for your GPU.

This is the most common cause for engineers starting out. Loading a 7B model in full 16-bit precision takes 14 GB. Add activations and optimizer states and you are at or past 24 GB before a single training step runs.

The fix is to enable 4-bit quantization. In your Axolotl config:

load_in_4bit: true adapter: qlora

This drops the base model footprint from 14 GB to roughly 4 to 5 GB and makes a 7B model trainable on a 24 GB GPU with room to spare.

Your sequence length is too high for your batch size.

Activation memory scales with sequence length times batch size. If your sequence length is 4096 and your micro batch size is 4, you are generating 16x more activation memory than a sequence length of 1024 with a batch size of 1.

Check your actual data. What is the 95th percentile length of your training examples? If most of your examples are under 1024 tokens, there is no reason to set sequence length to 4096. You are reserving memory for empty space.

In your Axolotl config:

sequence_length: 1024 # match to your actual data micro_batch_size: 2

Cutting sequence length is usually more effective than cutting batch size because it reduces activation memory directly. Try this before cutting batch size to 1.

Your batch size is still too high after reducing sequence length.

If you have already reduced sequence length and you are still hitting OOM, cut micro batch size. But do not just cut it to 1 and walk away. Compensate with gradient accumulation to keep your effective batch size reasonable.

micro_batch_size: 1 gradient_accumulation_steps: 8

This gives you an effective batch size of 8 while only holding 1 example worth of activations in memory at a time.

You are not using gradient checkpointing.

Gradient checkpointing trades compute time for memory. Instead of storing all activations for the backward pass, it recomputes them on the fly. This can cut activation memory by 50 to 70 percent at the cost of a 20 to 30 percent slower training run.

For memory-constrained setups this is almost always worth it.

gradient_checkpointing: true

Add this to your Axolotl config and rerun. If you were close to the memory limit, this usually clears it.

Your lora_r is too high.

Higher LoRA rank means more trainable parameters, which means more gradient and optimizer state memory. A lora_r of 256 on a 7B model adds meaningful memory overhead compared to a lora_r of 16 or 32.

If you set a high rank trying to improve results, try dropping it to 16 and seeing if the OOM clears. For most tasks the quality difference between rank 16 and rank 64 is small. The memory difference is not.

lora_r: 16

lora_alpha: 32

Memory fragmentation from a previous failed run.

If you ran a job that failed and immediately tried to run another one on the same instance without clearing memory, PyTorch may still be holding allocated memory from the previous run.

Restart the Python process or the instance entirely before your next run. This is a simple fix that people miss.

If you are in a notebook environment, restart the kernel. Do not just re-run the training cell.

The Order to Try Things

Work through these in order. Each one is faster to try than the next.

1. Enable 4-bit quantization if you are not already using it
2. Reduce sequence length to match your actual data
3. Enable gradient checkpointing
4. Cut micro batch size and add gradient accumulation steps to compensate
5. Reduce lora_r to 16 or 32
6. Restart the instance to clear fragmented memory

Most OOM errors in fine-tuning are resolved by step two or three. If you have worked through all six and are still hitting the error, your model size genuinely exceeds your GPU capacity and you need either a larger GPU or a multi-GPU setup.

One Quick Diagnostic Trick

Add this to the top of your training script before the training loop starts:

import torch print(f"GPU memory allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB") print(f"GPU memory reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")

Run it right after model loading and right before training starts. The difference between those two numbers tells you exactly how much memory the forward pass and optimizer initialization are consuming. That narrows down which fix applies to your situation.