From PPO to GRPO, Dr. GRPO, DAPO, GSPO, and DHPO

TL;DR

PPO learns a policy and a value function. GRPO removes the learned critic and compares responses to other responses for the same prompt. Dr. GRPO targets normalization-induced biases; DAPO combines several training fixes; GSPO moves importance weighting to the sequence level; and DHPO combines token- and sequence-level views. These names are not interchangeable?the correct choice depends on the failure being addressed.

PPO: The Expensive Baseline

PPO uses the probability ratio (r_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{old}(a_t|s_t)) and clips it around one. In LLM post-training, the actor is already large and the critic can approach the actor’s size, adding parameters, optimizer state, and a second estimation problem. The classical intuition remains essential; see the site’s actor-critic guide.

GRPO: Replace the Critic with a Group

For prompt (q), sample (G) completions and rewards (r_1,\ldots,r_G). A common group-relative advantage is

[ A_i=\frac{r_i-\operatorname{mean}(r_1,\ldots,r_G)}{\operatorname{std}(r_1,\ldots,r_G)+\epsilon}. ]

The group supplies a prompt-specific baseline. This saves the critic but couples learning quality to group diversity. If every reward is identical, every advantage is zero. GRPO also does not make rollouts cheap; it asks for several completions per prompt.

import torch

def group_advantage(rewards: torch.Tensor) -> torch.Tensor:
    mean = rewards.mean(dim=1, keepdim=True)
    std = rewards.std(dim=1, keepdim=True, unbiased=False)
    return (rewards - mean) / std.clamp_min(1e-4)

Dr. GRPO: Audit the Normalizers

?Understanding R1-Zero-Like Training? identifies biases caused by reward-standard-deviation scaling and response-length normalization. The proposed corrections remove those normalizers rather than treating the popular recipe as canonical. The practical lesson is broader: log both per-token and per-response objectives, because dividing by response length changes which completions dominate an update.

DAPO: A Systems Recipe, Not One Trick

DAPO combines asymmetric clipping, dynamic sampling, token-level policy-gradient loss, and explicit handling of overlong responses. Dynamic sampling removes prompts whose groups have no reward variation. Token-level aggregation prevents each response from receiving equal weight regardless of length. These choices are coupled, so a fair comparison changes one component at a time.

GSPO and DHPO: What Is the Unit of Correction?

Token-level importance ratios can vary wildly across a long sequence. GSPO defines the correction at sequence level, aligning the optimization unit with a sequence-level reward and improving stability in the reported large-scale training setting. DHPO proposes a hybrid: sequence-level control for global alignment and token-level ratios where local credit matters.

Choosing Honestly

Start with a maintained reference implementation, a deterministic verifier, and GRPO only when group sampling is affordable. Add dynamic sampling if zero-variance groups waste capacity. Investigate normalization before interpreting length changes as better reasoning. Use GSPO or a hybrid only when sequence-level correction addresses an observed instability; novelty is not evidence of suitability.

Key Learnings

1. Critic-free does not mean rollout-free or baseline-free.
2. Loss aggregation and normalization are algorithmic choices, not harmless implementation details.
3. Comparisons require the same model, prompts, sampling policy, verifier, and token budget.

References

1. Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms.
2. Shao, Z. et al. (2024). DeepSeekMath.
3. Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective.
4. Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale.
5. Zheng, C. et al. (2025). Group Sequence Policy Optimization.
6. Decoupled Hybrid Policy Optimization (2026 preprint).