Reinforcement Learning for LLM Alignment: From Reward Modeling to PPO-Based RLHF and DPO

TL;DR

Aligning an LLM to be helpful and safe is a reinforcement learning problem in disguise: the model is a policy, generating a response is a trajectory, and human preference is the reward signal — but that reward isn’t directly differentiable, so it has to be learned or reformulated. Reward modeling trains a model to score human preference from pairwise comparisons; PPO-based RLHF (the InstructGPT-style pipeline) uses that reward model to fine-tune the LLM with the PPO algorithm covered in the companion actor-critic article, constrained by a KL penalty against a reference policy; DPO shows the whole RL loop is mathematically unnecessary and collapses it into a single supervised loss; and GRPO keeps an RL formulation but removes PPO’s separate critic network, using group-relative reward comparisons instead. This is the algorithm family behind instruction-following chat models and reasoning-model training.

Problem Statement

There’s no differentiable loss function for “be helpful, honest, and harmless” — you can’t write is_good_response(x) as a loss you backprop through. What you can get is humans reliably comparing two candidate responses and saying which is better. Turning that comparative preference signal into a training update for an LLM is exactly the credit-assignment problem RL exists to solve, with the LLM’s own token-by-token generation process as the sequential decision process.

Reward Modeling: Turning Preferences Into a Scalar Reward

The first stage trains a separate reward model to predict human preference, using the Bradley-Terry model of pairwise comparison — given a prompt and two candidate responses, humans (or an AI judge) label which is preferred, and the reward model is trained so preferred responses score higher:

def reward_model_loss(reward_model, prompt, response_chosen, response_rejected):
    r_chosen = reward_model(prompt, response_chosen)
    r_rejected = reward_model(prompt, response_rejected)

    # Bradley-Terry: probability the chosen response is preferred, as a function of the score gap
    return -F.logsigmoid(r_chosen - r_rejected).mean()

This reward model is typically initialized from the same pretrained LLM (with a scalar output head), since it needs to understand language well enough to judge response quality, not just generate text.

PPO-Based RLHF: The Full Pipeline

With a reward model in hand, the LLM itself becomes the policy, and it’s fine-tuned with PPO (the clipped-objective actor-critic method from the companion article) to maximize the learned reward, with one critical addition — a KL-divergence penalty against a frozen reference copy of the model, preventing the policy from drifting so far in pursuit of reward that it degenerates into gibberish the reward model happens to score highly (reward hacking):

def rlhf_ppo_reward(reward_model, reference_policy, current_policy, prompt, response, kl_coef=0.1):
    task_reward = reward_model(prompt, response)

    log_prob_current = current_policy.log_prob(response, prompt)
    log_prob_reference = reference_policy.log_prob(response, prompt)
    kl_penalty = kl_coef * (log_prob_current - log_prob_reference)

    return task_reward - kl_penalty   # total reward fed into the PPO update

graph TD
    A[Pretrained LLM] --> B[Supervised Fine-Tuning - SFT]
    B --> C[Reward Model - trained on preference pairs]
    B --> D[Reference Policy - frozen SFT copy]
    B --> E[Policy - trainable, starts as SFT copy]
    E --> F[Generate Response]
    C --> G[Score Response]
    D --> H[KL Penalty vs Reference]
    G --> I[PPO Update]
    H --> I
    I --> E

    style E fill:#6366f1,color:#fff
    style I fill:#10b981,color:#fff

This three-stage pipeline — SFT, then reward modeling, then PPO fine-tuning against that reward with a KL constraint — is what InstructGPT and its successors popularized as “RLHF.”

DPO: Collapsing the RL Loop Into a Single Loss

Direct Preference Optimization (Rafailov et al., 2023) makes a striking observation: for the specific reward and KL-penalty structure used in RLHF, the optimal policy has a closed-form relationship to the reward model — which means you can substitute that relationship directly into the Bradley-Terry preference loss and get a loss function purely in terms of the policy’s own log-probabilities, with no separate reward model, no PPO, no sampling rollouts, and no RL training loop at all:

def dpo_loss(policy, reference_policy, prompt, response_chosen, response_rejected, beta=0.1):
    def log_ratio(model, response):
        return model.log_prob(response, prompt) - reference_policy.log_prob(response, prompt)

    chosen_ratio = log_ratio(policy, response_chosen)
    rejected_ratio = log_ratio(policy, response_rejected)

    return -F.logsigmoid(beta * (chosen_ratio - rejected_ratio)).mean()

DPO trains directly on the same preference pairs the reward model would have used, but as an ordinary supervised loss — dramatically simpler to implement and tune than the full PPO pipeline, at the cost of losing the ability to sample fresh, on-policy generations during training the way PPO does.

GRPO: RL Without a Separate Critic

GRPO (Group Relative Policy Optimization, used prominently in DeepSeek’s reasoning models) keeps the RL formulation and PPO’s clipped surrogate objective, but removes the need for a separate learned critic/value network — which is expensive to train and maintain at LLM scale. Instead, for each prompt it samples a group of multiple candidate responses, scores them all with the reward model, and uses each response’s reward relative to the group’s mean as the advantage directly:

def grpo_advantage(rewards_in_group: list[float]):
    rewards = torch.tensor(rewards_in_group)
    return (rewards - rewards.mean()) / (rewards.std() + 1e-8)   # advantage without a critic

This removes an entire network’s worth of training instability and memory cost compared to PPO’s actor-critic setup, at the cost of needing to sample multiple responses per prompt during training instead of just one.

Comparison Table

Applications

• Instruction-following chat models — the entire modern “assistant” behavior of production LLMs is shaped by exactly this family of techniques on top of a pretrained base model.
• Safety alignment — reward models trained on harmlessness preferences, combined with the KL-constrained PPO or DPO loop, are the primary mechanism for steering models away from harmful outputs.
• Reasoning model training — GRPO in particular has been used to train models that produce long chains of reasoning, where group-relative comparison of multiple sampled reasoning paths per problem provides the training signal.

Key Learnings

1. RLHF is a direct application of the actor-critic and policy-gradient ideas covered elsewhere on this site — the LLM literally is the policy network, and understanding PPO’s clipped objective and KL-constrained trust regions transfers directly to understanding why RLHF is stable (or unstable) in practice.
2. DPO’s core insight is that you don’t need the RL machinery at all for this specific problem structure — a closed-form relationship between the optimal RLHF policy and the reward model lets you skip straight to a supervised loss, which is a rare case of a full RL formulation turning out to be mathematically unnecessary.
3. The critic is often the most expensive and fragile part of PPO at LLM scale, which is exactly what GRPO’s group-relative advantage estimation is designed to eliminate — a reminder that at large enough scale, removing a component can matter as much as adding one.