Why Your Fine-Tuning Job Is Slower Than Expected (And Why GPU Utilization Is Below 50%)

TL;DR

A training job that is running slower than expected, or a GPU sitting at 30 to 50 percent utilization when it should be near 90 percent, is a signal that something in the training pipeline is not feeding the GPU work fast enough. This is not always an error. But it is always waste. This post covers the five most common causes of slow fine-tuning jobs and low GPU utilization, how to diagnose which one you have, and the specific fix for each.

You kicked off your training run. The loss is moving. Things seem to be working.

But the tokens per second is lower than the benchmarks you read suggested. And when you check nvidia-smi, GPU utilization is sitting at 35 percent.

A GPU at 35 percent utilization is not broken. But you are paying for 100 percent of it and getting a third of the throughput. Over a multi-hour training run, that is real money and real time.

Here is what is causing it and how to fix it.

Start Here: How to Actually See What Is Happening

Before you change anything, get a clear picture of where the bottleneck is.

Open two terminal windows. In the first, run:

watch -n 1 nvidia-smi

This shows GPU utilization, memory usage, and power draw updated every second. Watch the utilization column during training. Is it consistently low, or does it spike up between pauses?

If utilization spikes high briefly and then drops to near zero repeatedly, your GPU is waiting for data between steps. That is a dataloader bottleneck.

If utilization is consistently low and steady, your batch size is too small or your precision is wrong.

In the second terminal, watch your training logs for tokens per second or steps per second. Compare to your expected throughput. The gap tells you how much you are leaving on the table.

Cause One: The Dataloader Is the Bottleneck

This is the most common cause of low GPU utilization in fine-tuning and the most frequently missed.

Your GPU processes a batch in milliseconds. If your dataloader takes longer than that to prepare and load the next batch, the GPU sits idle waiting. From nvidia-smi this looks like utilization that spikes then drops, spikes then drops, in a repeating pattern.

Diagnose it:

Add timing to your training loop temporarily:

import time

for step, batch in enumerate(dataloader):
    load_start = time.time()
    # time how long the dataloader takes
    batch = {k: v.cuda() for k, v in batch.items()}
    load_time = time.time() - load_start

    train_start = time.time()
    # your training step
    train_time = time.time() - train_start

    if step % 10 == 0:
        print(f"Load: {load_time:.3f}s | Train: {train_time:.3f}s")

If load time is consistently higher than train time, the dataloader is your bottleneck.

Fix: increase num_workers

In your Axolotl config or your dataset config:

dataloader_num_workers: 4

This runs data preparation in parallel threads instead of the main process. Start with 4 and increase to 8 if the problem persists. Do not go above the number of CPU cores you have available.

Fix: enable dataloader pin_memory

dataloader_pin_memory: true

This pre-pins batches in CPU memory so GPU transfers are faster.

Cause Two: Batch Size Is Too Small

A batch size of 1 with no gradient accumulation means your GPU processes one example, updates weights, processes one example, updates weights. The overhead between steps is proportionally larger the smaller your batch is.

What it looks like:

GPU utilization is consistently low, not spiky. Tokens per second is well below what the GPU should be capable of.

Fix: increase micro batch size or add gradient accumulation

If memory allows, increase micro_batch_size:

micro_batch_size: 4

If you are memory constrained and cannot increase micro_batch_size without hitting OOM, use gradient accumulation to increase your effective batch size without using more memory:

micro_batch_size: 1

gradient_accumulation_steps: 8

This processes 8 batches before updating weights, giving the GPU more continuous work to do while keeping the per-step memory footprint small.

Cause Three: Wrong dtype or Precision Setting

Training in float32 is 2x to 4x slower than bfloat16 on modern NVIDIA GPUs, which have dedicated hardware for 16-bit operations. If your training is running in full 32-bit precision unintentionally, you are paying a large speed penalty for no quality benefit.

Diagnose it:

import torch
print(next(model.parameters()).dtype)

If this prints torch.float32 and you expected bfloat16, that is your problem.

Fix: set dtype explicitly in your Axolotl config

bf16: true

Or if your GPU does not support bfloat16 (older than Ampere architecture):

fp16: true

A100, H100, and RTX 30/40 series all support bfloat16. Use it. The speed improvement is significant and there is no quality loss for fine-tuning.

Cause Four: Sample Packing Is Off and Your Sequences Are Short

If your training examples are significantly shorter than your sequence_length setting, your GPU is spending time processing padding tokens that contribute nothing to the training signal. A batch of examples that average 256 tokens with a sequence_length of 2048 means 87 percent of each batch is padding.

Diagnose it:

Look at your dataset. What is the average length of your training examples? Compare it to your sequence_length setting.

Fix option one: reduce sequence_length to match your data

sequence_length: 512 # match to your actual data distribution

Fix option two: enable sample packing

Sample packing fills each sequence to the maximum length by concatenating multiple short examples end to end. This eliminates padding waste and dramatically improves GPU utilization on short-sequence datasets:

sample_packing: true

Note: sample packing changes how your data is tokenized. Test it on a short run first and confirm your loss curve looks normal before committing to a full training run.

Cause Five: Flash Attention Is Not Enabled

Standard attention is quadratic in sequence length. Flash Attention is a mathematically equivalent but much more memory-efficient and faster implementation. On long sequences it can be 2x to 4x faster.

Check if it is enabled:

python -c "from flash_attn import flash_attn_func; print('flash attention available')"

If this throws an import error, Flash Attention is not installed.

Install it:

pip install flash-attn --no-build-isolation

Enable it in your Axolotl config:

flash_attention: true

Flash Attention requires an Ampere or newer NVIDIA GPU (A100, A10G, RTX 30 series or newer). If you are on an older GPU it will not install successfully and you can skip this fix.

The Utilization Diagnostic in Order

Work through these checks before assuming you need a faster GPU.

One: watch nvidia-smi during training. Is utilization spiky or consistently low?

Two: if spiky, fix the dataloader. Increase num_workers and enable pin_memory.

Three: if consistently low, check your dtype. Enable bf16 if you are running float32.

Four: check your sequence length vs average example length. Enable sample packing or reduce sequence_length if you have significant padding.

Five: check if Flash Attention is installed and enabled. Install and enable it if your GPU supports it.

A GPU sitting at 30 percent utilization after all five of these changes is a GPU that genuinely needs a smaller model or a different approach. A GPU sitting at 30 percent utilization before any of these changes is a GPU waiting for you to feed it correctly.