What Are Warmup Steps in Fine-Tuning and Do They Actually Matter

TL;DR

Warmup steps are a learning rate scheduling technique that gradually increases the learning rate from zero to its target value at the start of training. The goal is to protect early training from large, potentially destabilizing gradient updates before the optimizer has built up reliable gradient statistics. This post explains what warmup steps actually do mechanically, when they matter for fine-tuning, when they can be skipped, and what values to use for the most common fine-tuning setups.

You are reading through an Axolotl config and you see:

warmup_steps: 10
# or sometimes
# warmup_ratio: 0.03

You set it because the example config had it. But do you actually need it? And if so, what should it be?

Here is the clear answer.

What Warmup Steps Actually Do

During training, the learning rate controls how large each weight update is. A high learning rate means large updates. A low learning rate means small, careful updates.

Warmup steps change the learning rate schedule at the very start of training. Instead of jumping immediately to your target learning rate, the scheduler starts at zero and linearly increases to the target value over the warmup period.

So if your target learning rate is 2e-4 and you have 100 warmup steps, the learning rate during those steps looks like this:

• Step 1: 2e-6
• Step 10: 2e-5
• Step 50: 1e-4
• Step 100: 2e-4 (full rate)
• Step 101 onward: whatever your main scheduler does

After warmup, the learning rate follows your chosen schedule. Linear decay, cosine decay, or constant, depending on your lr_scheduler_type setting.

Why Early Training Is Sensitive

At the very start of fine-tuning, the optimizer has no history. Optimizers like AdamW maintain running estimates of the gradient mean and variance for each parameter. These estimates take a few steps to warm up and become reliable.

In the early steps, gradient estimates are noisy and potentially inaccurate. If you hit those early steps with a full-size learning rate, the weight updates can be large and poorly directed. This can cause the model to make large, unstable moves in weight space before settling into productive learning.

Warmup gives the optimizer time to build up reliable statistics before applying full-size updates. By the time the learning rate reaches its target, the optimizer has seen enough data to estimate gradients more accurately.

For fine-tuning specifically, this matters more than for pre-training. In fine-tuning you are starting from a well-trained model. Large, poorly directed updates in the first few steps can damage the representations the base model learned before the optimizer stabilizes. Starting slow and ramping up protects those representations during the critical early steps.

When Warmup Steps Actually Matter

Short training runs with high learning rates.

If you are running only a few hundred steps with an aggressive learning rate, the warmup period is a significant fraction of your total training. Without it, the first steps have an outsized influence on where the model ends up.

When you see loss spikes in the early steps.

If your loss curve shows a sharp spike in the first 10 to 20 steps before settling into a downward trend, that is the optimizer overcorrecting on noisy early gradients. Adding or increasing warmup steps smooths this out.

When using AdamW or any adaptive optimizer.

AdamW, Adam, and similar optimizers maintain per-parameter gradient statistics that take several steps to become accurate. Warmup directly compensates for this initialization period.

When You Can Skip or Minimize Warmup

Very long training runs relative to warmup.

If you are training for 5,000 steps, 10 warmup steps is 0.2 percent of training. The effect is minimal either way. The warmup period needs to be a meaningful fraction of early training to matter.

Very low learning rates.

If your learning rate is already conservative, 1e-5 or lower, the risk of early instability is much lower. Warmup is less necessary.

When using learning rate schedulers that handle early steps natively.

Some schedulers like cosine with restarts or one-cycle policies handle the ramp-up in their own curve. In those cases, additional warmup steps may be redundant.

What Values to Use

There are two ways to set warmup in Axolotl: fixed steps or a ratio of total steps.

# Fixed number of steps
warmup_steps: 10

# Ratio of total training steps
warmup_ratio: 0.03

warmup_ratio is usually the better choice because it scales with your total training length. A 3 percent warmup on a 300-step run is 9 steps. On a 3,000-step run it is 90 steps. The warmup stays proportional regardless of how long your training is.

Recommended starting values:

For most QLoRA fine-tuning jobs: warmup_ratio of 0.03 to 0.05. This is 3 to 5 percent of total training steps.

For very short runs under 100 steps: warmup_steps of 5 to 10. A fixed count works better when the ratio would produce fewer than 5 steps.

For aggressive learning rates above 3e-4: increase warmup to 0.05 or higher. More warmup protects against instability at higher learning rates.

A working example for a typical 7B QLoRA run:

learning_rate: 0.0002
lr_scheduler_type: cosine
warmup_ratio: 0.03

This ramps from zero to 2e-4 over the first 3 percent of training, then follows a cosine decay to zero over the remaining 97 percent.

The Interaction With Learning Rate Schedulers

Warmup does not replace a learning rate scheduler. It is a phase at the beginning of whatever schedule you are using.

After warmup completes, the scheduler takes over. The most common choices in fine-tuning are:

linear: decays the learning rate linearly from the target to zero over the training run. Simple and predictable.

cosine: decays following a cosine curve, which starts fast and slows near the end. Generally produces slightly better results than linear for fine-tuning and is the most commonly recommended default.

constant: holds the learning rate at its target after warmup. No decay. Use this when you are running a very short test and do not want the learning rate to drop before you have seen useful signal.

For most fine-tuning jobs the recommendation is:

lr_scheduler_type: cosine
warmup_ratio: 0.03

This is the combination most fine-tuning practitioners have converged on as a reliable default. Start there and adjust only if your loss curve behavior tells you something specific is wrong.

The Short Answer

Do warmup steps matter?

Yes, for standard fine-tuning runs they are worth including. The cost is near zero. You are simply starting at a lower learning rate for the first few percent of training. The benefit is protection against early instability that can send your model in the wrong direction before the optimizer has reliable gradient estimates.

Use warmup_ratio of 0.03 with a cosine scheduler unless you have a specific reason to do otherwise. That is the default that works well across the widest range of fine-tuning tasks and model sizes.