TL;DR
A reward curve cannot tell you whether GRPO is learning robust behavior or exploiting its grader. Always inspect group reward variance, response length, entropy, KL divergence, clipping fraction, verifier outcomes, and representative raw completions together.
The Diagnostic Dashboard
| Signal | Warning | Likely response |
|---|---|---|
| Policy entropy | Fast fall toward zero | Reduce update pressure; audit sampling and bonuses |
| Response length | Growth without accuracy gain | Audit normalization and truncation reward |
| Group reward standard deviation | Many exact zeros | Resample/drop uninformative prompts |
| KL to reference | Sudden sustained rise | Check learning rate, clipping, and KL controller |
| Clip fraction | Persistently high | Updates are too large or data too stale |
| Train?held-out reward gap | Growing | Suspect verifier overfitting or contamination |
Entropy Collapse
When the policy becomes nearly deterministic, same-prompt groups lose diversity and the group baseline stops being useful. A higher temperature can mask rather than fix collapse. Compare token entropy before and after updates, stratified by prompt difficulty, and inspect whether a small set of phrases dominates completions.
Length Explosion
Longer answers can receive more opportunities to stumble onto a match, while per-response and per-token normalization assign them different update weights. Plot reward against length and accuracy against length. If reward rises while verified correctness does not, investigate the verifier and overlong-response policy rather than celebrating longer reasoning.
Zero-Variance Groups
If all (G) samples receive the same reward, standardized group advantages are zero. Report the fraction of sampled prompts with zero variance; do not hide it inside a batch mean. Dynamic sampling can replace these prompts, but repeated all-zero groups may instead reveal that the curriculum is too difficult.
def summarize_group(rewards, lengths):
return {
"reward_mean": rewards.mean().item(),
"reward_std": rewards.std(unbiased=False).item(),
"zero_variance": rewards.unique().numel() == 1,
"length_mean": lengths.float().mean().item(),
}
Reward Hacking and KL Blow-Up
Save raw outputs and verifier reason codes. A model can increase the scalar reward by exploiting formatting, parser fallbacks, or leaked tests. KL is also contextual: too little constraint permits destructive drift; too much prevents improvement. Log the measured KL used by the optimizer, not a differently aggregated dashboard approximation.
Reproducible Debugging Protocol
Keep model, dataset, seed set, prompt formatter, maximum tokens, and verifier version fixed. Change one intervention per run. Report distributions and confidence intervals rather than a single best checkpoint. This article intentionally provides a protocol, not fabricated benchmark results.
References
- Yu, Q. et al. (2025). DAPO.
- Liu, Z. et al. (2025). Understanding R1-Zero-Like Training.
- Zheng, C. et al. (2025). Group Sequence Policy Optimization.