What Is Gradient Accumulation and When Should You Use It

Abstract

Gradient accumulation is a training technique that lets you simulate a larger batch size than your GPU memory can hold at once. Instead of updating model weights after every batch, you accumulate gradients across several smaller batches and update once at the end. The result is that the model sees an effectively larger batch before each weight update, which often improves training stability, without requiring more GPU memory per step. This post explains exactly how it works, when it actually helps, when it does not, and how to configure it correctly.

You are setting up a training run. You want a batch size of 16 but your GPU runs out of memory at batch size 2.

The standard advice is to use gradient accumulation. Set micro_batch_size to 2 and gradient_accumulation_steps to 8. Now your effective batch size is 16.

That is the right advice. But most explanations stop there without explaining what is actually happening, when it matters, and when it does not. So you end up cargo-culting a setting without understanding what you are trading off.

Here is the full picture.

What Is Actually Happening

In standard training, the loop looks like this:

1. Load a batch of examples
2. Run the forward pass to compute predictions
3. Compute the loss
4. Run the backward pass to compute gradients
5. Update the model weights
6. Repeat

Each batch produces one weight update. The batch size controls how many examples the model sees before each update.

With gradient accumulation, the loop changes:

1. Load a small batch of examples
2. Run the forward pass
3. Compute the loss
4. Run the backward pass, but do not update weights yet. Accumulate the gradients.
5. Repeat steps 1 to 4 for N batches
6. After N batches, sum the accumulated gradients and update the weights once
7. Reset the gradients and repeat

The model weights are updated once every N batches instead of every batch. From the model's perspective, it is as if it processed a single batch of N times the size, because the gradient signal from all N batches is combined before the update.

Why This Saves Memory

GPU memory during training is consumed by four things: model weights, activations from the forward pass, gradients from the backward pass, and optimizer states.

The activations are the memory-intensive part. They scale with batch size because you need to hold the activations for every example in the batch in memory simultaneously during the backward pass.

With gradient accumulation, you process one small batch at a time. Each small batch has small activations. After the backward pass, you save the gradients but discard the activations. Then you process the next small batch, accumulate more gradients, discard those activations, and so on.

The gradients from each mini-batch are small and additive. They accumulate in memory but they take up far less space than the activations would for a large batch.

The result: you get the gradient signal of a large batch while only ever holding the activations of a small batch in memory at any one time.

How to Set It in Your Config

In Axolotl, gradient accumulation is controlled by two settings:

micro_batch_size: 2          # examples processed per GPU per step
gradient_accumulation_steps: 8   # steps before a weight update

Your effective batch size is:

effective_batch_size = micro_batch_size × gradient_accumulation_steps × number_of_GPUs

On a single GPU with the settings above, effective batch size is 2 × 8 = 16.

On two GPUs, it would be 2 × 8 × 2 = 32.

A typical starting point for 7B QLoRA training:

micro_batch_size: 2
gradient_accumulation_steps: 4

Effective batch size of 8. Adjust based on your available memory and how much you want to pay per step.

When Gradient Accumulation Actually Helps

When you are memory-constrained and want a larger effective batch size.

This is the primary use case. If your task benefits from larger batch sizes but you cannot fit them in memory, gradient accumulation is the right tool.

When training on very small datasets.

Smaller effective batch sizes can cause noisy gradient updates on small datasets because each batch is a less representative sample of the full training distribution. A larger effective batch size gives the model a more stable signal per update.

When you are seeing loss instability.

If your loss curve is spiky and erratic, increasing gradient accumulation steps can smooth it out. You are averaging gradient signal over more examples per update, which reduces the influence of any single unusual example.

When Gradient Accumulation Does Not Help

When your micro batch size is already large enough to fill your GPU.

If you have 24 GB of VRAM and your training step is only using 8 GB, you have headroom to increase micro_batch_size directly. Do that first. Processing larger actual batches is faster than accumulating across small ones because it reduces overhead between steps.

When training speed is critical and you have the memory.

Gradient accumulation adds N forward and backward passes per weight update instead of one combined pass. For large N, this can slow your training wall time compared to a single large batch that fits in memory. If you have memory to spare, use it.

When using certain normalization layers.

Batch normalization layers compute statistics over the actual batch at each step, not the accumulated effective batch. Gradient accumulation does not help batch norm see more examples per step. For fine-tuning transformers with layer normalization, this is not an issue. Layer norm computes statistics per example, not per batch.

The Tradeoff to Know

Gradient accumulation is not free. Each step is a separate forward and backward pass with its own overhead. If you accumulate over 16 steps, you run 16 separate forward and backward passes to produce one weight update. This is mathematically equivalent to one large batch update but computationally slower because of the per-step overhead.

For most fine-tuning workloads the difference is small. The memory savings are worth the overhead. But for jobs where raw training speed is the priority and memory is available, increasing micro batch size directly is always faster than equivalent gradient accumulation.

A Practical Starting Point

If you are unsure where to start, use this approach.

Set micro_batch_size to the largest value that runs without OOM. Run one step and check your GPU memory with nvidia-smi. If you are using less than 80 percent of available VRAM, increase micro_batch_size. If you are over 90 percent, reduce it.

Once micro_batch_size is set, choose gradient_accumulation_steps to get your target effective batch size. For most 7B QLoRA fine-tuning tasks, an effective batch size of 8 to 32 is a reasonable range. Start at 16 and adjust based on your loss curve behavior.

# Good starting point for a 7B QLoRA run on a 24GB GPU
micro_batch_size: 2
gradient_accumulation_steps: 8
# Effective batch size: 16