How to Fine-Tune Your First LLM Without Touching Infrastructure

TL;DR

Most fine-tuning guides start at step three. They assume you already have a GPU environment configured, CUDA installed, and a working framework setup. For someone doing this for the first time, that assumption means the guide is not actually a starting point. This post covers the full path for a first fine-tuning project: choosing a base model, preparing your dataset, writing a config, and running a training job without managing GPU infrastructure from scratch.

Here is the thing about most fine-tuning tutorials.

They show you the training code. They show you the config. They show you the loss curve. And they quietly assume that you already have a working environment to run all of it in.

If you do not, you spend the first two days before the actual tutorial even starts.

CUDA installation. Driver versions. PyTorch compatibility. Framework setup. None of it shows up in the tutorial because the person who wrote it already solved it. So they skipped it. And you are stuck on it.

This post does not skip it. It is written for someone doing this for the first time who wants to get from a dataset to a trained model without first becoming a GPU infrastructure expert.

Step One: Pick a Base Model

Your base model is the starting point. You are not training from scratch. You are taking a model that already knows how to generate text and teaching it to do your specific task well.

For a first project, use a 7B model. The most commonly used options right now are Mistral 7B, Llama 3.1 8B, and Qwen 2.5 7B. All three are freely available on Hugging Face, well-supported by every major fine-tuning framework, and fit comfortably on a single GPU with QLoRA.

Do not start with a 70B model. The cost to experiment is too high and the iteration loop is too slow for a first project. Start with 7B. You can always move up after you have a working baseline.

If you have a specific task in mind, look for a base model that is closest to your domain. A model pre-trained on code is a better starting point for coding tasks than a general instruction model. A model pre-trained on multilingual text is a better starting point for non-English tasks. You are fine-tuning, not creating. The closer the base model is to your task, the less work the fine-tuning needs to do.

Step Two: Build Your Dataset

You need at least 200 examples to see meaningful behavior change from fine-tuning. For most focused tasks, 500 to 2,000 high-quality examples is the right range for a first project.

What makes an example? Each training example should have an input and an output that represents the behavior you want. If you are building a support bot that responds in a specific tone, each example is a user message and the correct response. If you are building a document classifier, each example is a document and its label. If you are training a model to extract structured data, each example is raw text and the correctly formatted extraction.

The most common format for fine-tuning is alpaca JSONL. Each line looks like this:

jsonl

{"instruction": "Classify the sentiment of this review.", "input": "The product arrived late and was damaged.", "output": "Negative"}

A few things that matter more than quantity at this stage:

Your outputs should be consistent. If some are one sentence and some are five paragraphs for the same type of task, the model learns inconsistency. Decide what a good output looks like and make sure every example in your dataset matches that standard.

Your examples should cover the range of inputs you expect in production. If your model will receive questions phrased in different ways, include that variety in your training data.

Run a quick JSONL validation before you do anything else:

bash

python -c "import json; [json.loads(line) for line in open('your_data.jsonl')]"

If it runs without errors, your file is valid. If it throws an error, fix the malformed line before continuing.

Step Three: Write Your Config

If you are using Axolotl, your config is a YAML file that describes everything about your training job. Here is a minimal working config for a first QLoRA run on a 7B model:

base_model: mistralai/Mistral-7B-v0.1

model_type: MistralForCausalLM

tokenizer_type: LlamaTokenizer

load_in_4bit: true

adapter: qlora

lora_r: 16

lora_alpha: 32

lora_target_modules:

- q_proj

- v_proj

- k_proj

- o_proj

datasets:

- path: your_dataset.jsonl

type: alpaca

sequence_length: 2048

micro_batch_size: 2

num_epochs: 3

learning_rate: 0.0002

val_set_size: 0.05

output_dir: ./outputs

save_steps: 200

Change base_model to the model you chose in step one. Change the datasets path to your actual file. Leave everything else as is for your first run. These are sensible defaults that will get you a clean first training job.

Do not optimize yet. Your goal for the first run is a clean job that completes without errors. Optimization comes after you have a baseline.

Step Four: Run the Job Without Managing Infrastructure

This is the part most guides skip.

To run an Axolotl training job you need a machine with a GPU, CUDA installed, and all your Python dependencies set up correctly. Building that from scratch on a cloud instance takes 2 to 8 hours on a first attempt and produces an environment that is fragile and hard to reproduce.

There are two ways to avoid this.

Option one: use a pre-built Docker image. Axolotl publishes official Docker images with all dependencies pre-installed. If you are comfortable with Docker, you can pull the image and run your training job inside it without touching CUDA or driver installation.

bash

docker pull winglian/axolotl:main-latest

This works but still requires you to provision and manage a GPU instance yourself.

Option two: use a platform that handles the infrastructure layer for you. Heulistic lets you upload your Axolotl config, see the cost estimate for your job, and submit without provisioning an instance, installing CUDA, or managing anything below the config level. The platform handles instance selection, environment setup, and job execution. You interact with the training job, not the infrastructure underneath it.

For a first project this difference is significant. Instead of spending day one on environment setup, you spend day one on your dataset and your config. The parts that actually affect your model.

You can get started at heulistic.com.

Step Five: Evaluate Before You Ship

Your training run completed. The loss curve looks reasonable. Now do not skip the most important step.

Before you declare the model ready, compare its outputs against the base model on at least 20 to 30 inputs that represent your real use case. Give both models the same inputs. Look at the outputs side by side.

Does the fine-tuned model do what you trained it to do? Is the output style consistent? Does it handle edge cases reasonably?

If the answer is yes across your test inputs, you have a working model. If the answer is no or sometimes, you have a dataset or config issue to fix. That is not a failure. That is the fine-tuning loop working as intended. Adjust the data, rerun, and evaluate again.

What to Expect From Your First Project

Your first fine-tuning project will take longer than you expect. Not because fine-tuning is hard, but because you are building familiarity with the process at the same time as you are building the model.

Expect 1 to 3 days from dataset to a model worth evaluating. Expect 2 to 4 training runs before you have something solid. Expect at least one issue with your dataset format or config that you catch on the first run and fix before the second.

All of that is normal. Every experienced ML engineer has a version of the same first project story. The difference between the first project and the second is not just the model. It is that the second one goes faster because you have already solved the setup problems once.