How to Generate Synthetic Training Data When You Don't Have Enough Examples

TL;DR

Most teams trying to fine-tune a model do not have enough labeled training data. Real examples are expensive to collect, slow to annotate, or restricted by privacy requirements. Synthetic data generation solves this problem, but done poorly it produces datasets that inflate your example count without improving your model. Done well, research consistently shows that 500 to 2,000 filtered synthetic examples outperform tens of thousands of unfiltered ones. This post covers the three main approaches to generating synthetic training data, how to filter for quality, and the mistakes that produce datasets that look large but train poorly.

You have 150 real examples of the task you want your model to learn.

You know you need at least 500. Probably closer to 1,000 for the results you are after.

You could spend weeks collecting and annotating more real data. Or you could generate the rest synthetically in a day.

Synthetic data generation has become a standard part of fine-tuning workflows for a simple reason. It works. The research on this is consistent: a small set of high-quality real examples expanded thoughtfully with synthetic data outperforms a dataset built from real examples alone, as long as you filter for quality and keep the synthetic data distribution close to your real data.

The failure mode is generating thousands of synthetic examples without filtering and wondering why your model did not improve.

Here is how to do it right.

Start With Real Seed Examples

Before you generate anything, you need a seed dataset. These are your real examples. The ones you actually have.

Your seed examples matter more than your synthetic examples. They define the distribution your synthetic data should stay close to. They set the quality bar. They give the generating model concrete patterns to follow rather than general instructions.

Aim for at least 20 to 50 real seed examples before generating synthetically. Fewer than that and the generating model does not have enough signal to stay on task. More is better but 20 is workable.

The quality of your seeds is more important than the quantity. Five mediocre seed examples produce worse synthetic data than 20 excellent ones.

Approach One: Direct Expansion With a Stronger Model

This is the most straightforward approach. You have real examples. You ask a stronger model to generate more examples in the same format.

Here is a prompt structure that works:

import anthropic
import json

client = anthropic.Anthropic()

seed_examples = [
    {
        "instruction": "Classify the sentiment of this customer review.",
        "input": "The product arrived on time but the quality was disappointing.",
        "output": "Negative"
    },
    # ... your other seed examples
]

seed_text = "\n".join([json.dumps(e) for e in seed_examples[:5]])

prompt = f"""Here are examples of a sentiment classification task:

{seed_text}

Generate 10 new examples in the exact same JSON format. 
Follow these rules strictly:
- Instructions must be identical across all examples
- Inputs should be customer reviews of varying lengths and topics
- Outputs must be exactly one of: Positive, Negative, or Neutral
- Do not include any text outside the JSON objects
- Each example on its own line

Generate the examples now:"""

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=2000,
    messages=[{"role": "user", "content": prompt}]
)

print(message.content[0].text)

A few things that matter in the prompt:

Tell the model the exact format. Show it with real examples. Specify the constraints on outputs explicitly. And tell it not to include any text outside the JSON. That last instruction saves you parsing headaches.

Run this in batches. Generate 10 to 20 examples at a time rather than asking for 1,000 at once. Smaller batches stay more consistent with your seed examples.

Approach Two: Vary the Inputs, Keep the Task Fixed

If your task has a fixed output type, like classification, extraction, or structured transformation, the most reliable synthetic data approach is to vary the inputs while keeping the task and output format identical.

For a classification task, you can generate diverse inputs by category:

categories = [
    "electronics reviews", "food delivery reviews",
    "hotel reviews", "software reviews", "clothing reviews"
]

for category in categories:
    prompt = f"""Generate 5 customer reviews for {category} that would be classified as Negative sentiment.

Rules:
- Each review should be 1 to 3 sentences
- Reviews should feel realistic, not obviously AI-generated
- Vary the specific complaints and writing styles
- Output one review per line, no labels or formatting

Generate the reviews:"""
    # call your generating model here

Then pair each generated input with its known label. You already know the output because you controlled the generation. This is called gold-label generation and it eliminates the need to validate outputs individually.

This approach works well for classification, sentiment analysis, NER, and structured extraction tasks.

Approach Three: Self-Instruct Style Expansion

For instruction-following tasks where the input and output both vary, Self-Instruct is the most practical approach.

The idea is simple. You have seed instructions. You ask a model to generate new instructions that are meaningfully different from your existing ones, then generate responses for those new instructions.

existing_instructions = [e["instruction"] for e in seed_examples]

prompt = f"""Here are existing task instructions:
{chr(10).join(existing_instructions)}

Generate 5 new task instructions that:
- Are meaningfully different from the existing ones
- Test a different skill or knowledge area
- Are at a similar difficulty level
- Could realistically appear in a dataset for this domain

Output one instruction per line, nothing else:"""

# Generate instructions
# Then for each new instruction, generate a high-quality response

The key constraint is meaningfully different. If your generating model produces instructions that are semantically similar to your existing ones, you are creating apparent diversity with no real diversity. Your model learns the same patterns more times, not new patterns.

The Filtering Step You Cannot Skip

Generating examples is the easy part. Filtering them is where quality is actually determined.

Filter one: format check

Every generated example should parse correctly and match your expected format exactly. Run your JSONL validation script on all generated examples before anything else. Malformed examples cause silent training failures.

Filter two: length consistency

Generated outputs should match the length distribution of your real examples. If your real outputs average 50 words and your synthetic outputs average 200 words, your model will learn the wrong length distribution.

import json
import statistics

real_lengths = [len(e["output"].split()) for e in seed_examples]
mean_real = statistics.mean(real_lengths)
std_real = statistics.stdev(real_lengths)

filtered = []
for example in synthetic_examples:
    output_length = len(example["output"].split())
    if abs(output_length - mean_real) <= 2 * std_real:
        filtered.append(example)

Filter three: pattern matching for known bad outputs

The same filter you apply to real data applies here. Hedge phrases, refusal patterns, AI-sounding language. Generating models produce these even with careful prompting. Run your bad pattern filter on all synthetic outputs.

Filter four: manual spot check

Sample 50 synthetic examples and read them. You are looking for examples that are technically correct but feel off. Outputs that are too generic. Inputs that do not reflect realistic variation. Patterns that the generating model defaulted to across many examples.

If you see a pattern in your sample, it exists throughout your dataset. Fix the generation prompt and regenerate before training.

The Mistakes That Produce Bad Synthetic Datasets

Generating too many examples without filtering. A dataset of 50,000 unfiltered synthetic examples trains worse than 1,000 filtered ones. Generation is cheap. Filtering is where you earn the quality.

Not staying close to your real data distribution. If your real examples are short customer support responses and you generate long, detailed technical answers, your model learns the wrong pattern. Keep the style, length, and complexity of synthetic examples close to your real seed examples.

Using only synthetic data with no real examples. Pure synthetic datasets cause model collapse. The model memorizes patterns from the generating model rather than learning your task. Always mix real and synthetic. A minimum ratio of 10 to 20 percent real examples is a reasonable guideline.

Generating from a single prompt template. If every synthetic example was generated with the same prompt, your dataset has hidden uniformity even if the surface content varies. Use multiple prompt templates and generate across different model temperatures.

How Much Synthetic Data Is Enough

Research is consistent on this. More is not always better once you cross a quality threshold.

For a focused classification or extraction task: 500 to 1,000 filtered synthetic examples combined with your real seed examples is usually sufficient. More than that shows diminishing returns.

For a more complex instruction-following task: 1,000 to 3,000 filtered examples is a reasonable target. Quality filtering should remove 20 to 40 percent of what you generate. If you are filtering less than that, your filter is probably too permissive.

The test is always empirical. Train on your current dataset. Evaluate. Add more data only if the evaluation tells you to.