How to Write an Axolotl Config That Won't Waste Your GPU Budget

TL;DR

Axolotl is one of the most popular frameworks for fine-tuning large language models. But its config file has dozens of settings, and the wrong ones can silently waste compute, crash your run halfway through, or produce a model that looks trained but isn't. This post walks through the settings that actually matter, the ones you can safely ignore, and the specific mistakes that cause runaway jobs and wasted GPU hours.

You finally picked your base model. You chose QLoRA. You have a dataset ready.

And then you open an Axolotl config file for the first time.

It is a YAML file with 60 or 70 fields. Some are commented out. Some have values that look like they might matter. Some have values that definitely matter and nothing tells you which is which.

So you copy a config from a GitHub repo, tweak a few obvious things, and hit run. Sound familiar?

Here is the problem with that approach. A config copied from someone else's project was tuned for their model, their dataset, and their hardware. Not yours. And some of those defaults will quietly cost you money or quietly ruin your results before you even notice.

Let's fix that.

Start With the Non-Negotiables

There are five settings you must get right before anything else matters.

base_model is where you point Axolotl at your starting checkpoint. Get the path or Hugging Face model ID exactly right. A typo here downloads the wrong model and you won't always get an obvious error.

model_type and tokenizer_type need to match your base model. LlamaForCausalLM for Llama-family models, MistralForCausalLM for Mistral. Mismatching these causes subtle training failures that look like they're working.

datasets is where you define your data. The format field inside it matters a lot. Axolotl supports several dataset formats — alpaca, sharegpt, completion, and others. If your data is in alpaca format but you tell Axolotl it's sharegpt, the tokenization will be wrong. Your model will train on garbage and you'll get garbage out.

output_dir is where your checkpoints go. Make sure the path exists and has enough disk space. Running out of disk mid-training is a very common and very avoidable way to lose hours of compute.

The LoRA and QLoRA Settings That Actually Matter

If you're running QLoRA — which you probably should be if you're on a single GPU — you need these set correctly.

load_in_4bit: true turns on quantization. Without this, you're running LoRA on full precision weights and your memory usage goes up significantly.

adapter: qlora tells Axolotl to use QLoRA adapters. Set it to lora if you're not quantizing.

lora_r is the rank of your LoRA adapters. This controls how many parameters you're actually training. A value of 16 or 32 is a reasonable starting point for most tasks. Higher rank means more parameters, more memory, and potentially better results — but also higher cost. Do not set this to 256 on your first run.

lora_alpha is typically set to double your lora_r value. So if lora_r is 16, lora_alpha is 32. This is a widely used default and a safe starting point.

lora_target_modules tells Axolotl which layers to apply LoRA to. For most Llama and Mistral models, targeting q_proj, v_proj, k_proj, and o_proj is a solid default. Adding gate_proj, up_proj, and down_proj gives you more coverage at the cost of more parameters.

Sequence Length Is Where People Burn Money

sequence_length is one of the most impactful settings in the whole config. And it's also where I see the most waste.

GPU memory scales with sequence length. A sequence length of 4096 uses roughly four times the memory of 1024. If your training examples are mostly 512 tokens long, setting sequence_length to 4096 is burning memory on empty space.

Look at your actual data. Find the 95th percentile length of your examples. Set sequence_length to match that, not to the max the model supports. You will fit larger batches and train faster.

micro_batch_size works together with sequence length. The two together determine how much memory each step uses. If you're getting out-of-memory errors, try cutting sequence_length before you cut batch size. You'll often recover more memory that way.

The Settings You Can Safely Ignore at First

Axolotl has a lot of fields for things like special token handling, curriculum learning, and custom data pipelines. You do not need these on your first run or your second.

sample_packing is a useful optimization that packs multiple short examples into one sequence. But it changes how your data is processed and can interact with other settings in unexpected ways. Leave it off until your baseline is working.

gradient_checkpointing trades compute for memory. It lets you train with less GPU memory at the cost of a slower run. It is worth turning on if you are memory-constrained, but it is not a first-run priority.

wandb integration is great for tracking experiments. But if you haven't set up your Weights and Biases account yet, it will cause your run to fail at startup. Comment it out until you're ready for it.

The Mistakes That Cause Runaway Jobs

A runaway job is a training run that's clearly not working but keeps consuming compute anyway. Here's what causes them.

Wrong dataset format is the most common one. The loss drops fast at first and then plateaus near zero. Looks like great training. It's actually the model memorizing formatting tokens instead of learning from your data. Double-check your format field.

Learning rate too high causes loss spikes. If your loss curve looks like a heartbeat monitor instead of a smooth decline, your learning rate is too aggressive. A starting value of 2e-4 for QLoRA is reasonable. Go lower if you see instability.

No eval split means you have no idea if your model is actually improving or just memorizing. Always hold out at least 5 to 10 percent of your data for evaluation. Set val_set_size to 0.05 as a minimum.

Not setting save_steps means if your run crashes at step 900 of 1000, you lose everything. Set it to save every 200 steps at minimum. Disk space is cheap. Rerunning a job is not.

A Minimal Config That Works

Here is the shortest path to a working QLoRA run on a 7B model.

base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_target_modules:
  - q_proj
  - v_proj
  - k_proj
  - o_proj

datasets:
  - path: your_dataset
    type: alpaca

sequence_length: 2048
micro_batch_size: 2
num_epochs: 3
learning_rate: 0.0002

val_set_size: 0.05
output_dir: ./outputs
save_steps: 200

This is not a config optimized for peak performance. It is a config optimized for getting a clean first run with sensible defaults. From here you iterate.

The Point of All This

A config file is not just setup. It is a set of decisions about what you are training, how fast you are training it, and how much you are willing to spend to find out if it worked.

Every field you set without understanding is a decision you made by accident. And in fine-tuning, accidental decisions have a way of showing up as wasted GPU hours and models that almost work.

Start with the minimum. Get a clean run. Then change one thing at a time.

Get Started Training Your LLM