Safe and Constrained Reinforcement Learning: CPO, Lagrangian Methods, and Safe Exploration

TL;DR

Ordinary RL maximizes expected reward with no regard for how bad the worst outcomes along the way might be — fine for a game score, unacceptable for a robot that must never collide with a person. Safe RL reformulates the problem as a Constrained MDP: maximize reward subject to a hard limit on a separate cost signal. CPO (Constrained Policy Optimization) extends trust-region policy optimization to guarantee constraint satisfaction at every single update, not just on average. Lagrangian methods relax the hard constraint into a soft, adaptively-weighted penalty that’s simpler to implement and tune. Safety shielding takes a different approach entirely, filtering or overriding unsafe actions at the environment interface rather than relying on the policy to have learned safety. Used in robotics that operates near humans, industrial control systems, and autonomous vehicles, where “mostly safe” isn’t good enough.

Three Ways to Enforce a Constraint

graph TD
    A[Constrained MDP: reward + cost limit] --> B[CPO: hard constraint in the update itself]
    A --> C[Lagrangian Methods: soft, adaptive penalty]
    A --> D[Safety Shielding: filter unsafe actions at the interface]

    style D fill:#6366f1,color:#fff

Problem Statement

A reward function that penalizes unsafe behavior (e.g., a large negative reward for a collision) still allows the agent to trade off safety against reward during training and, worse, during early exploration before it has learned to avoid the bad outcome at all — the exact moments when an untrained policy is running on real hardware or near real people, and a single mistake can be irreversible. Safe RL treats safety as a separate constraint, not just another term folded into the reward, precisely because you don’t want the algorithm to be free to trade it away.

The Constrained MDP Formulation

Formally, alongside the usual reward r(s,a), a Constrained MDP defines a cost signal c(s,a) and requires the expected discounted cost to stay under a threshold d:

maximize   E[ Σ γ^t r(s_t, a_t) ]
subject to E[ Σ γ^t c(s_t, a_t) ] ≤ d

Every method below is a different strategy for solving this constrained optimization problem with a policy-gradient-style learner.

CPO: Constraint Satisfaction at Every Update

Constrained Policy Optimization (Achiam et al., 2017) extends TRPO’s trust-region idea (covered in the companion actor-critic article) with a second constraint alongside the usual KL trust region: the new policy’s expected cost must not exceed the threshold, approximated linearly around the current policy for tractability.

def cpo_step(policy, reward_advantage, cost_advantage, cost_limit, current_cost, kl_bound):
    # Approximate both objective and constraint linearly, trust region quadratically
    g = compute_policy_gradient(policy, reward_advantage)        # reward gradient
    b = compute_policy_gradient(policy, cost_advantage)          # cost gradient
    H = compute_fisher_information_matrix(policy)                # KL curvature

    # Solve the constrained quadratic program for the update direction
    step_direction = solve_cpo_dual(g, b, H, cost_limit - current_cost, kl_bound)
    return step_direction

If the linear approximation of the constraint is ever violated too severely by a proposed step, CPO falls back to a pure constraint-recovery step that ignores reward entirely and focuses only on getting back under the cost limit — a designed-in safety net for when the trust-region approximation itself breaks down.

Lagrangian Methods: Turning a Hard Constraint Into a Soft Penalty

CPO’s constrained trust-region solve is mathematically intricate. Lagrangian relaxation takes a simpler, widely-used approach: introduce a learned multiplier λ that adaptively penalizes constraint violation, turning the constrained problem into an unconstrained one that ordinary policy gradient or actor-critic methods can optimize directly.

def lagrangian_rl_step(policy_optimizer, lambda_multiplier, reward, cost, cost_limit, lambda_lr=0.01):
    # Policy optimizes reward minus the (currently weighted) cost penalty
    combined_objective = reward - lambda_multiplier * cost
    policy_optimizer.step(combined_objective)

    # Multiplier increases when constraint is violated, decreases when there's slack
    lambda_multiplier = max(0, lambda_multiplier + lambda_lr * (cost.mean() - cost_limit))
    return lambda_multiplier

As training progresses, λ automatically rises if the policy is violating the constraint (making the cost penalty increasingly dominant) and falls if the policy has comfortable safety margin (letting reward dominate again) — a self-tuning trade-off that’s far simpler to implement than CPO’s constrained trust region, at the cost of weaker guarantees about constraint satisfaction during any single update.

Safety Shielding: Enforcing Safety at the Interface, Not Just in the Policy

Both methods above still rely on the policy itself having learned to be safe, which is fragile during early training when the policy is still bad. Safety shielding instead sits between the policy and the environment: it maintains a (often hand-specified or formally verified) model of which actions are definitely unsafe in the current state, and overrides or filters out the policy’s proposed action before it ever reaches the environment.

def shielded_action(policy, state, safety_model):
    proposed_action = policy(state)
    if safety_model.is_safe(state, proposed_action):
        return proposed_action
    return safety_model.safe_fallback_action(state)   # override with a known-safe action

Because the shield’s safety guarantee doesn’t depend on what the policy has learned so far, it protects even a completely untrained, randomly-initialized policy from taking catastrophic actions during the very first exploration steps — a guarantee neither CPO nor Lagrangian methods can offer, since both rely on the policy itself gradually learning to respect the constraint.

Comparison Table

Applications

• Robotics operating near humans — collaborative robots and autonomous vehicles where a single unsafe action has real-world, potentially irreversible consequences.
• Industrial control systems — process control where certain states (over-pressure, over-temperature) must never be reached regardless of how much reward is on the table.
• Resource-constrained systems — power grid and data center control where operating limits (thermal, electrical) function as hard constraints alongside efficiency objectives.
• Any RL deployment during early training on real hardware — safety shielding specifically addresses the exploration-time risk that pure constrained optimization methods don’t fully solve.

Key Learnings

1. Treating safety as a separate constraint, not a reward penalty, is the core design decision of this entire field. A reward penalty is always tradeable against enough other reward; a genuine constraint is not supposed to be.
2. There’s a real trade-off between guarantee strength and implementation complexity. CPO offers the strongest per-update guarantees but is intricate to implement correctly; Lagrangian methods are simple and popular but only guarantee constraint satisfaction on average across training, not on any single step.
3. Safety during learning and safety of the final policy are different problems. CPO and Lagrangian methods both improve the learned policy’s safety over time, but only shielding directly protects against an early, still-bad policy taking a catastrophic action during exploration itself.

References

• Altman, E. (1999). Constrained Markov Decision Processes. CRC Press.
• Achiam, J. et al. (2017). Constrained Policy Optimization.
• Tessler, C., Mankowitz, D., Mannor, S. (2019). Reward Constrained Policy Optimization.