Preference-Based RL: Learning From Comparisons Instead of a Reward Function

TL;DR

Writing a correct reward function is often harder than it looks — even domain experts frequently can’t specify one that captures what they actually want without unintended side effects. Preference-based RL sidesteps the problem entirely: instead of a reward function, a human (or any oracle) is shown pairs of trajectory segments and simply says which one is better. A reward model is trained to be consistent with those comparisons, and any standard RL algorithm covered elsewhere on this site then optimizes against that learned reward. Active query selection makes this practical by choosing which trajectory pairs are most informative to ask about, rather than querying randomly. This is the general framework the RLHF pipeline covered in a companion article specializes for LLMs — preference-based RL predates and generalizes it to any RL domain, not just language.

Problem Statement

Consider training a robot to do a backflip. Writing a reward function for “good backflip” from scratch is genuinely hard — height, rotation speed, landing stability, and gracefulness all matter and trade off in ways that are difficult to specify numerically in advance, and a poorly specified reward risks the agent finding a technically-high-scoring but visually wrong solution (reward hacking). A human watching two candidate backflip attempts, by contrast, can usually say instantly which one looks better, even without being able to write down why as a formula. Preference-based RL turns that comparative judgment into a training signal.

The Core Pipeline: Comparisons to a Learned Reward Model

Deep RL from Human Preferences (Christiano et al., 2017) established the standard three-part loop: collect trajectory segments from the current policy, query a human for pairwise preferences between them, and train a reward model to be consistent with those preferences using the same Bradley-Terry formulation as the reward modeling stage of RLHF:

def preference_reward_model_loss(reward_model, segment_a, segment_b, human_prefers_a: bool):
    reward_a = reward_model(segment_a).sum()   # sum of per-timestep predicted rewards over the segment
    reward_b = reward_model(segment_b).sum()

    prob_a_preferred = torch.sigmoid(reward_a - reward_b)
    target = torch.tensor(1.0 if human_prefers_a else 0.0)
    return F.binary_cross_entropy(prob_a_preferred, target)

graph TD
    A[Current Policy] --> B[Generate Trajectory Segments]
    B --> C[Sample Pairs for Comparison]
    C --> D[Human/Oracle Preference Query]
    D --> E[Train Reward Model - Bradley-Terry Loss]
    E --> F[RL Training - any algorithm on this site]
    F --> A

    style E fill:#6366f1,color:#fff
    style F fill:#10b981,color:#fff

The policy is trained with an ordinary RL algorithm (the original paper used a policy-gradient method akin to the actor-critic family covered elsewhere on this site) against the learned reward model’s output, not against any hand-specified reward — the human never sees or writes a numeric reward at all, only comparative judgments.

Active Query Selection: Asking About the Right Pairs

Randomly sampling which trajectory pairs to query wastes human effort on comparisons that don’t teach the reward model much — two clearly-bad segments, for instance, provide little information regardless of which one is judged “less bad.” Active query selection instead prioritizes pairs the current reward model is most uncertain about:

def select_informative_pair(reward_model_ensemble, candidate_segment_pairs):
    uncertainties = []
    for seg_a, seg_b in candidate_segment_pairs:
        # Ensemble disagreement: how much do different reward models disagree on this pair's ranking?
        predictions = [torch.sigmoid(model(seg_a).sum() - model(seg_b).sum())
                       for model in reward_model_ensemble]
        uncertainty = torch.stack(predictions).std()
        uncertainties.append(uncertainty)

    most_uncertain_idx = torch.argmax(torch.stack(uncertainties))
    return candidate_segment_pairs[most_uncertain_idx]

This is the same ensemble-disagreement principle used in PETS’s uncertainty-aware planning (covered in the companion model-based RL article), applied here to decide which human queries are worth spending limited human attention on — a form of active learning specifically for the reward model rather than for the policy itself.

Connection to Dueling Bandits

The pairwise-comparison structure of preference queries connects directly to dueling bandits, a variant of the bandit problem (covered in the companion article) where, instead of pulling one arm and observing a reward, you choose two arms and observe only which one “won.” The same core challenge — extracting a ranking from noisy pairwise comparisons rather than absolute scores — underlies both fields, and much of the theory around efficiently querying comparisons originates in the dueling bandits literature before being adapted to full trajectory-level RL preferences.

Comparison Table

Applications

• Robotics behavior shaping — tasks like backflips, novel locomotion gaits, or any behavior where “looks right” is easy for a human to judge but hard to formalize as a reward.
• RLHF for LLM alignment — the reward-modeling stage of the RLHF pipeline (covered in a companion article) is a direct, large-scale application of exactly this framework, specialized to comparing text responses instead of trajectory segments.
• Any domain with an easy-to-judge, hard-to-specify objective — content moderation policies, dialogue quality, and creative generation tasks share the same “easy to compare, hard to score” structure.
• Reducing human labeling cost in RL research — active query selection specifically targets making human-in-the-loop RL training practical at all, since raw human comparison bandwidth is a hard constraint.

Key Learnings

1. Preference-based RL is a strictly more general framework than RLHF, not a competing one. RLHF for LLMs is this exact pipeline — reward modeling from comparisons, then RL against the learned reward — applied to one specific domain (text generation); understanding this article’s general framework is what makes the RLHF-specific companion article’s design choices legible as instances of a broader pattern.
2. The hard part isn’t training the reward model — it’s making the human query budget go far enough. Active, uncertainty-driven query selection is often the difference between a preference-based approach being practical or requiring an infeasible number of human comparisons.
3. Comparisons are often a more reliable signal than absolute scores, which is exactly why this framework exists. Humans are demonstrably better and more consistent at saying “A is better than B” than at assigning a stable, comparable numeric score to A and B independently — preference-based RL is built around exploiting that specific strength of human judgment.

References

• Yue, Y., Broder, J., Kleinberg, R., Joachims, T. (2012). The K-armed Dueling Bandits Problem. Journal of Computer and System Sciences, 78(5).
• Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences.