What Is DPO Fine-Tuning and When Should You Use It Instead of SFT

TL;DR

Supervised fine-tuning and direct preference optimization are two different approaches to improving a language model. SFT trains the model to imitate demonstrations of correct behavior. DPO trains the model by showing it pairs of responses and indicating which one is preferred. Each method is the right tool for different problems. This post explains what DPO actually does, how it differs from SFT mechanically, what kind of data it requires, and how to decide which method fits your use case.

Most fine-tuning starts with SFT. You collect examples of the behavior you want. You train the model to produce outputs that look like those examples. It works well for tasks with clear right answers.

But SFT has a ceiling.

When the goal is not just correctness but quality, helpfulness, tone, or alignment with human preferences, SFT runs into a fundamental limitation. It can teach the model to imitate. It cannot directly teach the model which of two reasonable responses is better.

That is what DPO was designed for.

What SFT Actually Does

Supervised fine-tuning is straightforward. You have input-output pairs. The model learns to produce outputs that match the training examples.

For classification, extraction, or structured tasks, SFT is often all you need. The right answer is well-defined. You show the model examples of correct outputs. It learns to produce them.

The limitation shows up on open-ended tasks. If you are training a chat assistant, a writing tool, or anything where the quality of the response matters more than a binary correct or incorrect, SFT gives you one good example per input. But it does not tell the model anything about why that example is good or what would make a response worse.

What DPO Does Differently

DPO stands for Direct Preference Optimization. It was introduced in a 2023 paper as a simpler alternative to RLHF, which required training a separate reward model.

The key difference from SFT is the data format. DPO does not train on individual examples of correct behavior. It trains on pairs.

Each training example has three parts: a prompt, a chosen response, and a rejected response. The chosen response is the one humans preferred. The rejected response is the one they did not.

{
  "prompt": "Explain what gradient descent is.",
  "chosen": "Gradient descent is an optimization algorithm that adjusts model parameters by moving in the direction that reduces the loss function. Think of it like walking downhill to find the lowest point in a valley.",
  "rejected": "Gradient descent is a mathematical optimization technique used in machine learning for minimizing objective functions through iterative parameter updates in the negative gradient direction of the cost function."
}

During training, DPO pushes the model to increase the probability of generating the chosen response and decrease the probability of generating the rejected response. Over many examples, the model learns to prefer the qualities that made chosen responses better than rejected ones.

What makes DPO powerful is that it learns from the contrast. The model does not just see what good looks like. It sees good and bad side by side and learns the difference.

The DPO Training Objective in Plain Language

Without getting into the math, here is what the DPO loss function is doing.

For each preference pair, it computes the ratio of how likely the model is to generate the chosen response versus the rejected response. It also computes the same ratio for the reference model, which is the base model before DPO training.

The objective is to increase this ratio. Make the model more likely to choose the chosen response relative to where it started, without drifting too far from the reference model's behavior.

The reference model acts as an anchor. It prevents the DPO-trained model from collapsing into always generating the same safe response or degrading the general capabilities it had before.

This is why DPO requires a reference model and SFT does not. SFT just needs examples. DPO needs examples and a baseline to compare against.

What DPO Is Good At

DPO is the right method when your goal is response quality improvement on open-ended tasks.

Improving helpfulness and response quality. If your SFT model gives technically correct responses that users still find unhelpful, DPO can teach it the difference between correct and genuinely useful.

Teaching the model to be more concise or more detailed. SFT on verbose examples makes verbose models. If you want to teach the model that shorter, clearer responses are better, DPO with preference pairs that contrast verbose and concise versions is more direct than SFT.

Reducing specific failure modes. If your model consistently makes a specific type of mistake, like being overly hedging or using formal language when casual language is preferred, DPO preference pairs that contrast the failure mode against better behavior can target that specific issue.

Alignment and safety. DPO is how many production models are tuned for safety after SFT. Preference pairs that contrast appropriate and inappropriate responses train the model to prefer safe behavior.

What DPO Is Not Good At

DPO is not a replacement for SFT on structured tasks.

If you need the model to always output JSON in a specific schema, SFT on correct JSON examples is the right approach. DPO does not help you teach the model a specific format. It helps you teach the model which of two reasonable outputs is better.

DPO also requires more effort to prepare data. SFT needs correct examples. DPO needs preference pairs, which means either human annotators comparing responses or a stronger model used as a judge to generate preference labels. If you do not have a reliable way to generate quality preference pairs, DPO data collection becomes the bottleneck.

The Typical Workflow: SFT First, Then DPO

In practice, most production fine-tuning workflows use SFT and DPO together in sequence. Not as alternatives.

Step one is SFT. Train the model on demonstrations of correct behavior for your task. This establishes the basic task capability and gives you a model that can produce reasonable outputs.

Step two is DPO. Take the SFT-trained model as the starting point. Collect preference pairs or generate them using a stronger model as a judge. Train with DPO to push the SFT model toward higher-quality responses.

The SFT step gets the model in the right territory. The DPO step refines quality within that territory.

Trying to use DPO without SFT on a base model is harder because the base model does not yet know your task. DPO works best when it is refining a model that already has the right general behavior.

How to Generate Preference Pairs Without Human Annotators

Human preference labeling is expensive and slow. For many teams it is not feasible at scale. Here are the practical alternatives.

Model-as-judge. Generate two responses to each prompt using your SFT model with different sampling temperatures or configurations. Then use a stronger model like GPT-4 to evaluate which response is better according to your criteria. This is the most common approach for teams that cannot afford human annotation.

Rejection sampling. Generate multiple responses per prompt. Use a scoring function, which might be a classifier, a heuristic, or another model, to score each response. The highest-scoring response becomes chosen and a lower-scoring response becomes rejected.

Targeted contrast generation. Deliberately generate responses that exhibit a specific failure mode you want to fix, for example responses that are too verbose, too hedging, or too formal. Pair these as rejected responses against your SFT model's better outputs as chosen responses.

The Decision in Plain Terms

Start with SFT. It is simpler, requires less data preparation, and solves most structured task fine-tuning problems completely.

Add DPO when your SFT model produces correct outputs that are still not as good as you need them to be. When the problem is quality, helpfulness, or preference rather than correctness, DPO is the tool designed for that gap.

If you are unsure whether you need DPO, you probably do not yet. Build an SFT baseline first. Evaluate it honestly. If the outputs are correct but the quality is still not where it needs to be after iterating on your data, that is the signal to add DPO.