RLVR: Reinforcement Learning with Verifiable Rewards, From DeepSeekMath to Production

TL;DR

Reinforcement learning with verifiable rewards (RLVR) replaces a learned preference model with a checker whose verdict can be reproduced. A math answer can be normalized and compared, code can be run against tests, and structured output can be validated against a schema. This makes the signal cheaper and harder to drift than an LLM judge, but only for tasks whose success can actually be specified.

RLVR does not remove reward design. It moves reward design into the verifier, where parsing errors, leaked tests, partial credit, and exploitable edge cases become the important engineering problems. It therefore complements, rather than replaces, the broader RLHF pipeline.

From Preferences to Checks

In preference-based RL, a reward model learns which response humans prefer. Optimizing against that proxy can expose blind spots in the model. RLVR instead defines a function (V(x,y)\rightarrow r), where (x) is the task and (y) is the response. For exact-answer tasks, (r\in{0,1}); richer environments may return test pass rates or rubric components.

The useful distinction is not ?automatic versus human.? It is verifiable versus judgeable. ?Does this program pass an isolated test suite?? is verifiable. ?Is this explanation elegant?? remains judgeable. Production post-training commonly combines both.

A Defensive Math Verifier

from fractions import Fraction
import re

def normalized_number(text: str) -> Fraction:
    candidate = text.strip().lower().replace(",", "")
    boxed = re.findall(r"\\boxed\{([^{}]+)\}", candidate)
    if boxed:
        candidate = boxed[-1]
    candidate = candidate.replace("%", "/100")
    if not re.fullmatch(r"[-+]?\d+(?:\.\d+)?(?:/\d+)?", candidate):
        raise ValueError("response is not a supported numeric answer")
    return Fraction(candidate)

def verify(response: str, expected: str) -> float:
    try:
        return float(normalized_number(response) == normalized_number(expected))
    except (ValueError, ZeroDivisionError):
        return 0.0

This deliberately accepts 0.5, 1/2, and 50% as equivalent while rejecting text that merely contains the expected number. A real verifier must also set input limits, isolate untrusted code, conceal tests, and record a reason code alongside the scalar reward.

The Training Loop

For each prompt, sample several responses, verify each one, convert rewards into advantages, and update the policy under a KL constraint. GRPO and its descendants are popular because group-relative comparisons avoid a second critic model. Sparse binary rewards create a curriculum problem: prompts that every sample fails give little useful discrimination, while prompts that every sample solves are already exhausted.

Failure Modes

Parser mismatch: correct reasoning receives zero because the answer extractor fails.
Verifier gaming: the model exploits a test harness, edits tests, or triggers a permissive fallback.
Contamination: benchmark answers or hidden tests leak into training data.
Outcome-only credit: the final verdict does not identify which reasoning step failed.
Specification ceiling: correctness can be checked, but helpfulness, safety, and style cannot be completely reduced to a deterministic predicate.

Keep verifier versions immutable, run adversarial cases before training, save raw outputs, and evaluate on separately authored tests. Never execute model-generated code on the host process.

Key Learnings

RLVR is powerful where correctness is cheaply reproducible, not wherever a model can emit a score.
The verifier is part of the environment and must be threat-modeled like production code.
Hybrid reward systems are unavoidable when a task mixes objective correctness with subjective quality.

References

Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
Lambert, N. et al. (2024). T?lu 3: Pushing Frontiers in Open Language Model Post-Training.
Lightman, H. et al. (2023). Let’s Verify Step by Step.