Dataset Format for Fine-Tuning: JSONL, Data Quality, and the Mistakes That Kill Results

TL;DR

Your dataset is the single biggest variable in fine-tuning. A great config and the right method can't save you from bad data. But bad data isn't always obvious. It doesn't always look broken. This post covers JSONL structure, the two dataset formats engineers use most, the quality signals that separate useful data from data that produces models that almost work, and the mistakes that are easy to miss until it's too late.

Here's something most people don't want to hear.

You can have the perfect config. The right base model. The right method. And still get a model that doesn't work. Not because of anything in your training setup — but because of what's in your dataset.

Data problems are the hardest fine-tuning problems to debug. They don't always throw errors. Sometimes the training run completes cleanly. The loss curve looks fine. And then you test the model and something is just off. Responses that almost make sense. Behavior that's close but not quite right.

That's what bad data looks like from the outside. Let's talk about what it looks like from the inside, and how to avoid it.

Start With the Format

Every fine-tuning framework — Axolotl, TRL, Unsloth — expects your data in JSONL. That's JSON Lines. One JSON object per line, no trailing commas, no wrapping array around the whole thing.

This is what a valid JSONL file looks like:

{"instruction": "Summarize this in one sentence.", "input": "The quick brown fox...", "output": "A fox jumped over a dog."}
{"instruction": "Translate to Spanish.", "input": "Hello, how are you?", "output": "Hola, ¿cómo estás?"}

Each line is its own complete example. The file has no opening bracket, no closing bracket, no commas between lines. Just one object per line, end of story.

If your JSONL file has syntax errors, most frameworks will either crash immediately or silently skip the broken lines. The second case is worse because you lose training examples without knowing it.

Run a quick validation before every training job:

python -c "import json; [json.loads(line) for line in open('your_data.jsonl')]"


If it runs without errors, your file is valid JSON. That's your minimum check.

Alpaca vs ShareGPT: Pick the Right Format

There are two formats you'll use for almost every fine-tuning job. Alpaca and ShareGPT. They're not interchangeable and picking the wrong one is one of the most common mistakes I see.

Alpaca format is for single-turn instruction following. Each example has an instruction, an optional input, and an output.

{
"instruction": "Classify the sentiment of this text.",
"input": "I loved the product but the delivery was terrible.",
"output": "Mixed"
}

Use alpaca when your task is structured and single-turn. Summarization, classification, extraction, transformation tasks. One input, one output, done.

ShareGPT format is for multi-turn conversations. Each example is a list of turns between a human and an assistant.

{
"conversations": [
{"from": "human", "value": "What's the capital of France?"},
{"from": "gpt", "value": "Paris."},
{"from": "human", "value": "And what's the population?"},
{"from": "gpt", "value": "About 2.1 million in the city proper."}
]
}

Use ShareGPT when you're training a chatbot, a coding assistant, or anything where context across turns matters.

The mistake people make is using alpaca format when their task is actually conversational, or using ShareGPT when their task is single-turn. The model trains on whatever format you give it. But its behavior at inference time depends on how it was trained. Format mismatch produces models that respond strangely to real inputs even though the loss looked fine during training.

How Much Data Do You Actually Need?

Less than you think. And more than you have.

For QLoRA and LoRA fine-tuning, you can get solid results with 500 to 5,000 high-quality examples for a well-defined task. Classification, summarization, extraction — these don't need massive datasets. What they need is clean, consistent, representative data.

For full fine-tuning or for tasks that require the model to learn a new domain from scratch, you need significantly more. Think tens of thousands of examples minimum.

But here's what trips people up. They hear "500 examples is enough" and they scrape together 500 examples from wherever they can find them. Some from internal docs. Some from GPT-4 outputs. Some from a dataset they found on Hugging Face that's kind of related. And they wonder why the model is inconsistent.

500 high-quality, consistent, on-task examples will outperform 5,000 mixed-quality examples every single time. The model learns patterns from your data. If your data has mixed patterns, you'll get a model with mixed behavior.

The Quality Signals That Actually Matter

You can't eyeball 2,000 examples. But you can spot quality problems at scale if you know what to look for.

Consistency of output style. If some of your outputs are one sentence and some are five paragraphs for the same type of task, the model will learn that both are acceptable. That produces inconsistent inference behavior. Decide what the right output style is and make sure your whole dataset matches it.

Instruction coverage. Does your dataset cover the range of inputs your model will see in production? If you train on formal English and deploy to users writing casually, the model will struggle. Sample from the real distribution wherever possible.

Noise in the output field. This is the one that kills results quietly. Output fields that have errors, inconsistencies, or formatting problems teach the model to reproduce those same errors. A dataset generated by a weaker model without quality filtering will have this problem. Filter your outputs before training.

Duplicates. Duplicate examples cause the model to overfit to those specific cases. They also inflate your apparent dataset size. Deduplicate before training. A simple hash-based dedup on the output field catches most of it.

Length outliers. Examples that are dramatically longer or shorter than the rest cause training instability. Set a minimum and maximum length and filter anything outside that range.

The Mistakes That Are Easy to Miss

Leaking eval examples into training. If any of your validation set examples appear in your training set, your eval metrics are meaningless. Your model will look like it's performing great on eval because it memorized those examples. This is more common than people admit and is usually caused by sloppy dataset splits on deduplicated data.

Inconsistent field names. Alpaca format expects instruction, input, and output. If your data has prompt and response instead, and your Axolotl config says alpaca, the framework might silently train on empty fields. Check that your field names match your declared format exactly.

System prompts in some examples and not others. If some of your ShareGPT examples have a system turn and others don't, your model will learn inconsistent behavior around system prompts. Decide whether your model uses a system prompt and be consistent across the entire dataset.

GPT-generated data without filtering. Generating your dataset with a stronger model is a legitimate and effective approach. But unfiltered GPT output has hedging language, refusals on edge cases, and formatting patterns that you probably don't want your model to learn. Run a quality filter or at minimum do a manual review of a random sample before you train on it.

A Simple Pre-Training Checklist

Before every training run, run through these five checks.

One: validate your JSONL syntax. Two: confirm your field names match your declared format. Three: check for duplicates and remove them. Four: review a random sample of 50 examples manually. Five: verify your train and validation sets don't overlap.

This takes 20 minutes. It saves you from the most expensive class of fine-tuning failure — the one where training completes successfully and the model still doesn't work.