TL;DR
Reward systems differ along two independent axes: where feedback is assigned (final outcome or intermediate process) and how it is produced (learned model, deterministic rule, human label, or model judge). Calling all four ?reward models? hides the engineering trade-offs.
A Useful Taxonomy
| Signal | Strength | Main risk |
|---|---|---|
| Deterministic verifier | Reproducible and cheap | Narrow specification; exploitable checker |
| Learned preference model | Handles subjective qualities | Distribution shift and proxy optimization |
| LLM judge | Flexible, little training setup | Bias, inconsistency, self-preference |
| Process reward model | Better local credit | Expensive/noisy step labels |
Outcome supervision scores the final answer. Process supervision scores intermediate reasoning steps. A process score can improve credit assignment, but only if ?step correctness? is meaningful and labels are trustworthy. It can also reward verbose decompositions or penalize valid unconventional reasoning.
Training a Pairwise Reward Model
Given chosen and rejected responses with scalar outputs (r_w,r_l), a Bradley?Terry objective is -log(sigmoid(r_w-r_l)). Accuracy on held-out pairs is necessary but insufficient: RL actively searches for inputs that maximize the model, including regions absent from its validation set.
import torch.nn.functional as F
def pairwise_reward_loss(chosen_reward, rejected_reward):
return -F.logsigmoid(chosen_reward - rejected_reward).mean()
Production Design
Version reward components independently, log their raw values before aggregation, test calibration on new policy outputs, and retain adversarial holdouts. Use deterministic RLVR where possible, learned feedback where judgment is genuinely required, and human review for consequential ambiguity.
Do not interpret a weighted sum as a stable objective when components have different variances. Normalize and monitor each channel, then check whether improving the combined score harms any constituent objective.
References
- Christiano, P. et al. (2017). Deep Reinforcement Learning from Human Preferences.
- Ouyang, L. et al. (2022). Training Language Models to Follow Instructions with Human Feedback.
- Lightman, H. et al. (2023). Let’s Verify Step by Step.
- Wang, P. et al. (2023). Math-Shepherd.