How to Resume a Failed Fine-Tuning Job From a Checkpoint

TL;DR

A training job that fails halfway through is not automatically a complete loss. If checkpoints were saved during training, you can resume from the last saved point and continue without retraining from scratch. This post explains how checkpointing works in Axolotl, how to configure it so you always have something to resume from, how to resume a failed job correctly, and the specific mistakes that cause resume to fail silently or produce incorrect results.

Here is a situation you want to avoid.

A 6-hour training job fails at hour 5. You check the logs. The process died from an OOM error. Or the spot instance was reclaimed. Or a network connection dropped during a checkpoint upload.

If you did not set up checkpointing, you restart from step 1. You just lost 5 hours of compute.

If you did set up checkpointing, you find the last saved checkpoint and resume from there. You lose 20 minutes of compute at most. Everything before the last save is intact.

Checkpointing is not an optional nicety. It is the difference between a failed run costing you an hour versus a day.

Here is how to do it right.

How Checkpointing Works in Axolotl

A checkpoint is a snapshot of the model weights and optimizer state at a specific training step. When training resumes from a checkpoint, it restores the model to that exact state and continues from that step forward.

There are two things saved in a checkpoint: the model weights (for LoRA this is the adapter weights, not the full model) and the optimizer state. The optimizer state includes the momentum and variance estimates for each trainable parameter that AdamW uses to compute weight updates. Without restoring the optimizer state, resuming from a checkpoint produces different training dynamics than if the job had never failed. The loss curve will look different and the final model will be different.

Axolotl saves both. When you resume correctly, training picks up exactly where it left off.

Configuring Checkpointing Before You Start

Add these settings to your Axolotl config before your first run. Not after the job fails.

# Save a checkpoint every N steps

save_steps: 100

# How many checkpoints to keep on disk

save_total_limit: 3

# Where checkpoints are saved output_dir: ./outputs

save_steps controls how frequently checkpoints are written. Every 100 steps means the most you can ever lose from a failed job is 100 steps of compute. For a 1-hour training run that saves every 100 steps, a failure at the worst possible moment costs you a few minutes of recompute.

Setting save_steps too low has a cost. Writing a checkpoint takes time, especially for larger models. If your total training is 500 steps and you save every 10, you are spending significant time on checkpoint writes relative to training. A good starting point is to save often enough that a failure does not cost you more than 10 to 15 minutes of compute.

save_total_limit controls how many checkpoints are kept on disk at once. Setting it to 3 means Axolotl keeps the 3 most recent checkpoints and deletes older ones automatically. This prevents your disk from filling up on long runs. If you want to keep all checkpoints for evaluation purposes, remove this setting or set it to a large number.

output_dir is where checkpoints are written. Make sure this path has enough disk space. A checkpoint for a LoRA adapter is much smaller than for a full model, typically a few hundred MB rather than tens of GB. But for full fine-tuning, checkpoints can be large and disk space planning matters.

How to Resume From a Checkpoint in Axolotl

When your job fails and you want to resume, add one setting to your config:

resume_from_checkpoint: true

This tells Axolotl to find the most recent checkpoint in your output_dir and resume from there. It restores the model weights, the optimizer state, and the training step counter. Training continues exactly where it left off.

If you want to resume from a specific checkpoint rather than the most recent one:

resume_from_checkpoint: ./outputs/checkpoint-5400

Point it at the checkpoint directory for the specific step you want to resume from.

Then resubmit the job with the same config, same dataset, and the resume setting added. Do not change anything else. The whole point of resuming is continuity. Changing the learning rate, the batch size, or the dataset between a checkpoint and a resume produces inconsistent training that is hard to interpret.

What the Checkpoint Directory Contains

When you look inside a checkpoint directory, you will see files like these:

checkpoint-5400/

├── adapter_config.json

├── adapter_model.safetensors

├── optimizer.pt

├── scheduler.pt

├── trainer_state.json

├── training_args.bin

└── rng_state.pth

adapter_model.safetensors contains the LoRA adapter weights at that checkpoint step.

optimizer.pt contains the AdamW optimizer state. This is what allows resuming to continue with the same learning dynamics. If this file is missing or corrupted, resuming will produce different results.

trainer_state.json contains the training step count, the loss history, and the evaluation history. Axolotl uses this to know which step to resume from.

rng_state.pth contains the random number generator state. Restoring this ensures data sampling continues from the same point, which matters for reproducibility.

If any of these files are missing, the checkpoint is incomplete and resuming from it will either fail with an error or produce unexpected results. Check that all files are present before attempting a resume.

The Mistakes That Cause Silent Resume Failures

Changing the dataset between checkpoint and resume.

If you shuffle your dataset after saving a checkpoint, the training examples the model sees after resuming are different from what it would have seen if the job had never failed. The model effectively trains on some examples twice and others not at all. This produces a different final model than an uninterrupted run.

If you need to change your dataset, start a new run from scratch rather than resuming.

Changing hyperparameters between checkpoint and resume.

Resuming with a different learning rate or batch size is not the same as training with those settings from the start. The optimizer state was built up under the original settings. Introducing a different learning rate mid-training produces inconsistent dynamics.

If you want to experiment with different hyperparameters, treat the checkpoint as a starting point for a new experiment, not a resume of the old one.

Resuming without restoring the optimizer state.

Some resume workflows load the adapter weights but not the optimizer state. The model starts from the right weights but the optimizer resets to its initial state. This is equivalent to starting fine-tuning from a pre-trained checkpoint, not from mid-run. Loss will initially drop again before climbing back to where it was, and the final model will differ from what an uninterrupted run would have produced.

Axolotl handles this correctly when you use resume_from_checkpoint. But if you are manually loading weights in a custom training script, make sure you are also restoring the optimizer state.

Resuming from a checkpoint that was saved during evaluation.

If your checkpoint was saved at an eval step rather than a training step, the trainer state may reflect the eval step count rather than the training step count. Axolotl handles this correctly in most cases but it is worth checking trainer_state.json to confirm the step count before resuming.

On Managed Platforms

If you are running on a platform that handles infrastructure for you, like Heulistic, checkpointing still matters but the workflow is slightly different.

Heulistic automatically uploads checkpoints to S3 during training. You saw this in the real training log from the previous post: checkpoints uploaded at step 17,500 and step 17,551, with older checkpoints pruned automatically. If a job fails, the last uploaded checkpoint is available.

When you resubmit a job on Heulistic, the platform handles restoring the checkpoint to the new instance before training resumes. You do not need to manually copy checkpoint files between instances or configure S3 paths. The config setting and the platform handle the rest.

This is one of the concrete ways managed infrastructure removes overhead from the fine-tuning workflow. Fault tolerance is built in, not bolted on.

The Minimum Config for Safe Training

If you take nothing else from this post, add these three lines to every training config you write:

save_steps: 100

save_total_limit: 3

output_dir: ./outputs

You will never lose more than 100 steps of compute to a failed job. That is the whole point.