How to Split Your Dataset for Fine-Tuning: Train, Validation, and What Most People Get Wrong

TL;DR

Every fine-tuning guide tells you to split your dataset into training and validation sets. Most of them tell you to use an 80/20 split and move on. What they do not cover is why the split matters, how to do it in a way that produces evaluation metrics you can actually trust, the common mistakes that create data leakage between train and validation, and when the standard 80/20 split is the wrong choice for your specific dataset. This post covers all of that.

Splitting your dataset sounds like the boring part. It is two lines of code in most frameworks. You do it once and forget about it.

But a bad dataset split is one of the most reliable ways to produce evaluation metrics that look great and a model that does not work in production.

The problem is not the split itself. It is what happens before and after the split, and whether the validation set actually measures what you think it measures.

Here is how to do it right.

What the Validation Set Is Actually For

Your training set is what the model learns from. Your validation set is what you use to measure how well that learning is generalizing.

The key word is generalizing. The validation set is only useful if it contains examples the model has not seen during training. If any validation examples appear in the training set, the model has memorized those specific cases. Your validation metrics show performance on memorized data, not on new inputs. That is a meaningless number and it will not predict production performance.

This seems obvious. But data leakage between train and validation sets is one of the most common and least-diagnosed problems in fine-tuning projects.

The Standard Split and When It Is Right

The most common split is 80 percent training, 20 percent validation. For a dataset of 1,000 examples that means 800 training examples and 200 validation examples.

This is a reasonable default for medium-sized datasets where you have enough examples that 20 percent gives you a meaningful validation set. 200 examples is enough to get a stable estimate of model performance on most tasks.

For smaller datasets the math changes. If you have 200 examples total, a 20 percent validation split gives you 40 validation examples. That is small enough that your validation metrics can swing significantly based on which 40 examples happen to be selected. A few hard examples in the validation set can make the metrics look worse than the model actually is. A few easy examples can make them look better.

For larger datasets the math also changes. If you have 50,000 examples, a 20 percent validation split gives you 10,000 validation examples. You do not need 10,000 examples to get a reliable estimate of model performance. 500 to 1,000 is usually sufficient. Keeping 10,000 examples out of training is wasting 20 percent of your data for no additional benefit.

Choosing the Right Split Size

Here is a practical framework based on dataset size.

Under 500 examples: Use a 90/10 split. You need as many training examples as you can get. 50 validation examples is tight but workable. Consider supplementing with synthetic data if your validation set feels too small to trust.

500 to 5,000 examples: The 80/20 split works well here. 100 to 1,000 validation examples gives you a reliable estimate without sacrificing too much training data.

5,000 to 50,000 examples: Use a fixed validation set size of 500 to 1,000 examples rather than a percentage. Put the rest in training. A 99/1 split on a 50,000 example dataset gives you 500 validation examples and 49,500 training examples. That is better than holding out 10,000.

Over 50,000 examples: Same principle. Fix the validation set at 1,000 to 2,000 examples. Put everything else in training.

In Axolotl you set this with val_set_size:

# Percentage-based
val_set_size: 0.1   # 10% validation

# Fixed size (set as a fraction that gives you the right count)
# For 10,000 examples and 500 validation:
val_set_size: 0.05  # 5% = 500 examples

The Order of Operations That Prevents Leakage

This is where most people make the mistake that creates data leakage.

The wrong order:

1. Collect data
2. Deduplicate
3. Split into train and validation

The problem: deduplication after splitting can create overlap. If two near-duplicate examples end up on different sides of the split, and you then deduplicate, you may remove one from training but not from validation. The validation example is now nearly identical to a training example the model saw. Your metrics are measuring near-memorization.

The right order:

1. Collect data
2. Deduplicate the full dataset
3. Split into train and validation

Deduplicate first on the complete dataset. Then split. This guarantees that no example in your validation set has a near-duplicate in your training set.

import json
import hashlib
from sklearn.model_selection import train_test_split

# Load all examples
all_examples = []
with open("your_data.jsonl") as f:
    for line in f:
        all_examples.append(json.loads(line))

# Deduplicate first
seen = set()
deduped = []
for example in all_examples:
    key = hashlib.md5(
        (example.get("instruction", "") + example.get("output", "")).encode()
    ).hexdigest()
    if key not in seen:
        seen.add(key)
        deduped.append(example)

print(f"Removed {len(all_examples) - len(deduped)} duplicates")

# Then split
train_examples, val_examples = train_test_split(
    deduped,
    test_size=0.1,
    random_state=42  # set a seed for reproducibility
)

# Write to files
with open("train.jsonl", "w") as f:
    for example in train_examples:
        f.write(json.dumps(example) + "\n")

with open("val.jsonl", "w") as f:
    for example in val_examples:
        f.write(json.dumps(example) + "\n")

Always set random_state to a fixed value. A reproducible split means that if you add more data later and rebuild the split, you can verify that your validation set is consistent with previous runs.

The Case for Stratified Splitting

For classification tasks, a random split can produce a validation set that does not represent your class distribution. If you have 10 classes and 200 examples per class, a random split might end up with 30 examples of one class and 10 of another in the validation set. Your validation accuracy is now being driven partly by how many examples of each class were randomly assigned.

Stratified splitting preserves class proportions across both splits:

from sklearn.model_selection import train_test_split

labels = [example["output"] for example in deduped]

train_examples, val_examples = train_test_split(
    deduped,
    test_size=0.1,
    stratify=labels,
    random_state=42
)

Use stratified splitting whenever you have a classification task with more than two classes or any class imbalance in your dataset.

Do You Need a Test Set Too

A test set is a third split, held out entirely and only used once at the very end to measure final model performance before shipping.

The validation set is used during training to monitor overfitting and compare checkpoints. Because you look at it multiple times during the training process, your model indirectly adapts to it. You might make config changes or dataset changes specifically to improve validation metrics. That means your validation metrics are not a perfectly unbiased estimate of production performance.

A test set that you never look at until the final evaluation gives you that unbiased estimate.

For most fine-tuning projects at small to medium scale, a dedicated test set is worth having. The practical minimum is 100 examples that you do not look at until you are ready to make a final go or no-go decision on the model.

In practice, many teams skip the test set for early iterations and only add it when the project is close to production. That is reasonable. But do not use your validation metrics as your final production performance estimate without acknowledging that they may be slightly optimistic.

Verify Your Split Before Training

Add this check to your pre-training checklist. Run it after splitting, before you start the first training job.

import json
import hashlib

def hash_example(example):
    return hashlib.md5(
        (example.get("instruction", "") + example.get("output", "")).encode()
    ).hexdigest()

train_hashes = set()
with open("train.jsonl") as f:
    for line in f:
        train_hashes.add(hash_example(json.loads(line)))

overlap = 0
with open("val.jsonl") as f:
    for line in f:
        if hash_example(json.loads(line)) in train_hashes:
            overlap += 1

if overlap > 0:
    print(f"WARNING: {overlap} validation examples found in training set")
    print("Rebuild your split from the deduplicated dataset")
else:
    print("No overlap detected. Split looks clean.")

If this check passes, your validation set is measuring what you think it is measuring. If it fails, your evaluation metrics are not trustworthy, and you need to rebuild the split.