← Back to all posts
GDPOMulti-Objective RLReward Design

GDPO and Multi-Reward RL: Why Weighted Sums Break

How scale and variance let one reward dominate, and how per-reward normalization changes multi-objective policy optimization.

O Ohara University · · 14 min read

TL;DR

An assistant may need to be correct, concise, safe, and well formatted. Adding those rewards with fixed weights looks reasonable, but the component with the largest scale or variance can dominate the policy gradient. Group-wise normalization per reward preserves the relative signal before combination; it does not eliminate genuine conflicts among objectives.

Why the Weighted Sum Is Unstable

Let (r=\sum_k w_kr_k). A unit change in a binary correctness reward and a unit change in a judge score need not represent comparable evidence. Their variances also change during training: correctness may saturate while a style judge remains noisy. The effective priorities therefore drift even when weights stay fixed.

def normalized_multi_reward(reward_matrix, weights, eps=1e-4):
    # shape: prompts x samples x reward_components
    mean = reward_matrix.mean(dim=1, keepdim=True)
    std = reward_matrix.std(dim=1, keepdim=True, unbiased=False)
    normalized = (reward_matrix - mean) / std.clamp_min(eps)
    return (normalized * weights).sum(dim=-1)

This sketch must define behavior for zero-variance components. Silently dividing by epsilon can make dashboards appear healthy while a channel contributes no ranking information.

Relation to Classical Multi-Objective RL

The site’s multi-objective RL guide explains the deeper limitation: scalarization selects one trade-off on a Pareto frontier. Normalization fixes scale mismatch, not normative disagreement. Safety constraints may need to be enforced as constraints rather than paid for with enough helpfulness reward.

Evaluation Checklist

Report each raw reward, its variance, pairwise correlations, the combined advantage, and held-out task metrics. Ablate one reward at a time. Inspect examples where components disagree. A method has not solved multi-objective alignment merely because its aggregate reward increased.

References

  1. Roijers, D. et al. (2013). A Survey of Multi-Objective Sequential Decision-Making.
  2. Hayes, C. et al. (2022). A Practical Guide to Multi-Objective Reinforcement Learning and Planning.
  3. Shao, Z. et al. (2024). DeepSeekMath.