What Is GRPO and How Is It Different From PPO

TL;DR

PPO and GRPO are both reinforcement learning methods for fine-tuning language models. PPO, Proximal Policy Optimization, was the dominant approach for RLHF until recently. GRPO, Group Relative Policy Optimization, was introduced in the DeepSeek-R1 paper and has largely displaced PPO for most LLM fine-tuning use cases because it is simpler, uses less memory, and produces comparable or better results. This post explains what each method does, why GRPO became the preferred approach, and when you would actually use reinforcement learning fine-tuning versus simpler methods like SFT or DPO.

If you have been paying attention to the LLM fine-tuning space over the past year, you have seen GRPO show up everywhere. In papers. In framework changelogs. In community discussions about DeepSeek and reasoning models.

But a lot of the explanations assume you already know what PPO is and why GRPO is different.

Here is the version that starts from the beginning.

Why Reinforcement Learning Is Involved at All

SFT and DPO both learn from data you have collected ahead of time. You prepare examples or preference pairs, run training, and the model learns from that fixed dataset.

Reinforcement learning fine-tuning works differently. Instead of learning from a static dataset, the model generates responses during training, those responses are evaluated by a reward signal, and the model's weights are updated based on that reward. The model learns from its own outputs rather than from demonstrations you prepared in advance.

This has an important implication. If you have a reliable reward signal, RL fine-tuning can push a model to discover behaviors that were not present in any training demonstration. It can optimize for outcomes, not just imitation.

The trade-off is complexity. RL fine-tuning is harder to set up, less stable than SFT or DPO, and requires a reward function that actually captures what you want. But for tasks where the goal is clear and measurable, like math correctness, code that passes tests, or responses that score highly on a defined rubric, RL fine-tuning can produce results that SFT and DPO cannot match.

What PPO Does

PPO, Proximal Policy Optimization, became the standard RL fine-tuning method for LLMs through its use in InstructGPT and ChatGPT's RLHF pipeline.

In PPO, the setup involves four separate models running simultaneously:

The policy model is the model being trained. It generates responses.

The reward model is a separate model trained to score responses based on human preferences. It provides the reward signal.

The value model, also called the critic, estimates the expected future reward from any given state. It is used to compute a quantity called the advantage, which measures how much better or worse a response was than expected.

The reference model is a frozen copy of the policy model at the start of training. It is used to compute a penalty that prevents the policy from drifting too far from its starting point.

All four models need to be loaded in memory simultaneously during training. For a 7B model, this means four times the memory of a single model. That is a significant constraint.

PPO also requires careful hyperparameter tuning. The training dynamics are sensitive to the balance between exploration and exploitation, the reward clipping, and the frequency of policy updates. Getting PPO stable enough to train reliably takes experience.

What GRPO Does Differently

GRPO, Group Relative Policy Optimization, was introduced in the DeepSeek-R1 paper in early 2025. It achieves similar results to PPO with a significantly simpler setup.

The key innovation is how GRPO estimates the advantage. Instead of a separate value model that estimates expected future reward, GRPO generates a group of responses for each prompt, scores all of them with the reward function, and uses the relative differences within that group as the advantage signal.

If the model generates 8 responses for a prompt and response 3 scored higher than average, response 3 gets a positive advantage. If response 7 scored lower than average, it gets a negative advantage. The model learns to produce more responses like the high-scoring ones and fewer like the low-scoring ones.

This eliminates the need for the value model entirely. Instead of four models, GRPO requires two. The policy model and the reference model. Memory requirements roughly halve compared to PPO.

The reward model can also be simpler in many GRPO setups. For tasks with verifiable outcomes like math or code, the reward is computed directly from whether the answer is correct. No separate trained reward model needed. The reward function is a rule.

Why GRPO Spread So Quickly

The DeepSeek-R1 paper demonstrated that GRPO could train models to exhibit chain-of-thought reasoning, long internal reasoning traces, at a fraction of the cost of PPO-based approaches. The DeepSeek models trained with GRPO matched or exceeded much larger models on reasoning benchmarks.

This landed at a moment when the fine-tuning community was already frustrated with PPO's complexity and memory demands. GRPO offered comparable results with a simpler setup. Frameworks updated quickly. By mid-2025, TRL had full GRPO support. Axolotl followed. Unsloth added it shortly after.

By 2026, GRPO is the default starting point for RL fine-tuning tasks across the open-source community.

The Data Format for GRPO

SFT uses input-output pairs. DPO uses preference pairs. GRPO uses prompts plus a reward function.

Your training data for GRPO is just prompts. The model generates responses during training. The reward function evaluates those responses. There is no fixed output to imitate.

For math fine-tuning, a common reward function checks whether the model's final answer matches the correct answer:

def math_reward(prompt, response, correct_answer):
    # Extract the final answer from the response
    if correct_answer in response:
        return 1.0
    else:
        return 0.0

For code fine-tuning, the reward function runs the generated code against test cases:

def code_reward(prompt, response, test_cases):
    # Extract code from response and run tests
    score = 0
    for test in test_cases:
        if passes_test(response, test):
            score += 1
    return score / len(test_cases)

For tasks without a verifiable ground truth, you can use a judge model to score responses, similar to the model-as-judge approach used in DPO data generation.

When to Actually Use GRPO or PPO

This is the most important section and the one most explanations skip.

Both GRPO and PPO are reinforcement learning methods. They are more complex to set up than SFT or DPO, less stable during training, and require a reward function that genuinely captures what you want. They are not the right starting point for most fine-tuning tasks.

Use GRPO when your task has a verifiable outcome and you have exhausted what SFT and DPO can give you.

Math reasoning is the canonical example. The reward is binary: the answer is correct or it is not. SFT on correct solutions teaches the model to imitate the right answer. GRPO teaches the model to reason toward the right answer by rewarding the outcome rather than the process.

Code generation is another strong case. Tests either pass or they do not. GRPO can optimize directly for passing tests rather than imitating code that looks like it would pass tests.

Instruction following quality, when measured by a reliable judge model, is a weaker case but still valid. If you have a judge model that reliably distinguishes better from worse responses and DPO is not closing the quality gap you need, GRPO with a judge reward is worth trying.

Do not reach for GRPO when SFT would work. The added complexity of RL training, the instability, the reward function engineering, and the longer debugging cycles are real costs. Pay them only when the reward signal gives you something SFT and DPO cannot.

The Practical Starting Point

If you want to try GRPO for the first time, TRL has the most accessible implementation:

from trl import GRPOConfig, GRPOTrainer

config = GRPOConfig(
    output_dir="./grpo-output",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    num_generations=8,       # responses generated per prompt
    learning_rate=1e-5,
    max_new_tokens=512,
)

trainer = GRPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    reward_funcs=your_reward_function,
)

trainer.train()

The num_generations parameter is what makes GRPO work. It controls how many responses are generated per prompt to compute the group-relative advantage. More generations give a more stable advantage estimate at the cost of more compute. 4 to 8 is a reasonable starting range.

The Short Summary

PPO was the standard RL fine-tuning method. It required four models in memory and was complex to stabilize. GRPO achieves comparable results with two models and a simpler advantage estimation approach. The DeepSeek-R1 paper showed GRPO could train strong reasoning models at significantly lower cost, and the method quickly became the community default.

Both are reinforcement learning methods for tasks where the goal is measurable and the reward signal is reliable. For everything else, SFT and DPO are simpler, faster to iterate on, and easier to debug.

Start with SFT. Add DPO if quality needs improvement. Reach for GRPO only when your task has a verifiable reward and the simpler methods have hit their ceiling.