← Back to all posts
Reinforcement LearningMulti-Objective RLPareto Optimality

Multi-Objective Reinforcement Learning: Pareto Fronts and Reward Scalarization

How RL handles competing objectives that can't be collapsed into one number — scalarization methods that reduce multiple objectives to a single reward, and Pareto-front methods that learn a whole family of trade-off policies at once — with code and applications.

O Ohara University · · 11 min read

TL;DR

Real-world problems rarely have one objective — a delivery robot cares about speed and energy use and safety, and no single number naturally captures all three without someone deciding how to trade them off. Scalarization methods sidestep the problem by combining multiple objectives into one weighted reward before handing it to any standard RL algorithm covered elsewhere on this site. Pareto-front methods instead learn an entire family of policies spanning the trade-off curve, so the trade-off can be chosen after training rather than baked in ahead of time. Used in robotics balancing competing physical constraints, recommendation systems balancing engagement against diversity, and any resource-allocation problem with genuinely competing goals.

Picking a Trade-off vs. Learning the Whole Frontier

graph LR
    A[Multiple Competing Objectives] --> B[Linear Scalarization: fixed weights before training]
    A --> C[Non-Linear Scalarization: broader Pareto coverage]
    A --> D[Preference-Conditioned Policy: pick trade-off at deployment]

    style D fill:#6366f1,color:#fff

Problem Statement

Given objectives r₁, r₂, ..., rₖ that can’t be improved simultaneously past a certain point — more speed costs more energy — there’s no single “best” policy, only a set of policies where improving one objective necessarily worsens another. That set is called the Pareto front. Every technique in this article is a strategy for either picking one point on that front ahead of time (scalarization) or learning to represent the whole front at once.

Linear Scalarization: The Simple Default

The most common approach: pick weights and combine objectives into a single scalar reward, then apply any ordinary single-objective RL algorithm unchanged.

def scalarized_reward(objective_values: list[float], weights: list[float]):
    return sum(w * r for w, r in zip(weights, objective_values))

# Train an ordinary agent (e.g., PPO from the companion actor-critic article) on this scalar reward
def train_with_fixed_weights(env, agent, weights):
    for episode in range(n_episodes):
        objectives = env.step_and_collect_objectives()
        reward = scalarized_reward(objectives, weights)
        agent.update(reward)

This is simple and lets you reuse any single-objective algorithm without modification, but it has a real limitation: linear scalarization can only ever reach points on the convex hull of the Pareto front — trade-off points that require a non-convex combination of objectives are structurally unreachable no matter how the weights are tuned.

Non-Linear Scalarization and Constraint-Based Formulations

To reach non-convex parts of the Pareto front, non-linear scalarization functions (e.g., Chebyshev scalarization, which minimizes the maximum weighted deviation from an ideal point rather than a weighted sum) are used instead:

def chebyshev_scalarized_reward(objective_values, weights, ideal_point):
    weighted_deviations = [w * abs(ideal - r) for w, r, ideal in zip(weights, objective_values, ideal_point)]
    return -max(weighted_deviations)   # minimize the worst weighted deviation from the ideal

An alternative framing treats all but one objective as constraints rather than reward terms — maximize speed subject to energy use staying under a threshold — which connects directly to the constrained-MDP formulation covered in the companion safe RL article, just with the “safety” cost replaced by a competing objective.

Pareto-Front Methods: Learning the Whole Trade-off at Once

Rather than committing to one set of weights before training, Pareto-front methods learn a single policy conditioned on a preference vector, so the trade-off can be selected at deployment time by simply choosing what preference to feed in:

class PreferenceConditionedPolicy(nn.Module):
    def forward(self, state, preference_weights):
        # Preference vector is part of the input, not baked into a fixed scalar reward
        combined_input = torch.cat([state, preference_weights], dim=-1)
        return self.network(combined_input)

def train_pareto_conditioned(env, policy, preference_distribution):
    for episode in range(n_episodes):
        preference = preference_distribution.sample()   # sample a random trade-off point each episode
        objectives = env.rollout(policy, preference)
        reward = scalarized_reward(objectives, preference)
        policy.update(reward, condition=preference)

By training across many randomly sampled preference vectors rather than one fixed weighting, the resulting network learns to interpolate across the Pareto front — at deployment, a user or downstream system can pick any trade-off point without retraining, simply by feeding in a different preference vector.

Comparison Table

MethodTrade-off DecidedCoverage of Pareto FrontRetraining Needed to Change Trade-off?
Linear ScalarizationBefore training (fixed weights)Convex hull onlyYes
Non-Linear ScalarizationBefore training (fixed weights/ideal point)Broader, including non-convex regionsYes
Preference-Conditioned PolicyAt deployment timeFull front, if training preferences covered it wellNo

Applications

Key Learnings

  1. There is no single “optimal” policy when objectives genuinely compete — only a frontier of non-dominated trade-offs. Every technique here is a strategy for either picking a point on that frontier ahead of time or representing the whole frontier at once.
  2. Linear scalarization’s blind spot (the convex hull limitation) is a real, not theoretical, problem — plenty of practically interesting trade-off points are structurally unreachable by any weighting of a linear combination, which is exactly why non-linear scalarization and Pareto-front methods exist.
  3. Preference-conditioning turns a training-time decision into a deployment-time one, which is valuable whenever the “right” trade-off depends on context that isn’t known until after the policy is trained — a different user, a different operating regime, a different regulatory environment.

References