Explainable RL: Making Learned Policies Inspectable

TL;DR

Every deep RL method covered elsewhere on this site produces a policy that’s a large neural network — accurate, but opaque: it’s hard to know why it chose a given action, which matters enormously before deploying a policy in a safety-critical, regulated, or high-stakes setting. Saliency methods highlight which parts of the input state most influenced a decision, adapting techniques from supervised deep learning interpretability. Policy distillation (notably VIPER) trains an inherently interpretable model — a decision tree — to imitate a trained neural policy closely enough to inspect and, in some cases, formally verify. Reward decomposition breaks a single scalar Q-value into separate, individually meaningful components, so “why did the agent do that” can be answered in terms of which specific objective it was weighing most heavily. Used for safety validation, debugging, regulatory compliance, and building trust before deploying an RL policy in the real world.

Three Ways to Open the Black Box

graph TD
    A[Trained Neural Policy] --> B[Saliency Maps: which input mattered]
    A --> C[Policy Distillation - VIPER: inspectable decision tree]
    A --> D[Reward Decomposition: per-objective Q-values]

    style C fill:#6366f1,color:#fff

Problem Statement

A neural network policy that performs well in evaluation can still fail in ways that weren’t in the test distribution, and without any way to inspect why it makes the decisions it does, catching those failure modes before deployment — or diagnosing them after an incident — is far harder than it needs to be. This matters more, not less, as RL moves into higher-stakes domains: the healthcare, robotics, and financial applications mentioned throughout the other articles on this site are exactly the settings where “the network said so” is not an acceptable justification on its own.

Saliency Maps: Which Part of the State Mattered?

Adapting gradient- and perturbation-based attribution techniques from supervised deep learning, saliency methods highlight which regions of a state (e.g., which pixels of an image observation) most influenced the policy’s chosen action:

def perturbation_saliency(policy, state, action_taken, patch_size=8):
    baseline_q = policy.q_value(state, action_taken)
    saliency_map = np.zeros(state.shape[:2])

    for y in range(0, state.shape[0], patch_size):
        for x in range(0, state.shape[1], patch_size):
            perturbed_state = state.copy()
            perturbed_state[y:y+patch_size, x:x+patch_size] = blur_patch(state, y, x, patch_size)
            perturbed_q = policy.q_value(perturbed_state, action_taken)
            saliency_map[y:y+patch_size, x:x+patch_size] = abs(baseline_q - perturbed_q)

    return saliency_map

Blurring or masking each region and measuring how much the policy’s Q-value for the chosen action changes gives a direct, model-agnostic measure of that region’s importance — a large drop means the policy was relying heavily on that part of the state, which is often enough on its own to catch a policy that’s “cheating” by relying on an unintended visual cue (e.g., a game score display) rather than the actual gameplay-relevant content.

Policy Distillation: VIPER and Verifiable Trees

Saliency maps explain individual decisions but don’t summarize the overall policy logic. VIPER (Verifiable RL via Policy Extraction, Bastani et al., 2018) instead trains an inherently interpretable decision tree to imitate the trained neural policy, using a DAgger-style approach (covered in the companion imitation learning article) so the tree is trained on states the neural policy actually visits, not just a fixed offline dataset:

def viper_distillation_step(neural_policy, tree_policy, env, dataset, max_depth=8):
    # DAgger-style: roll out the CURRENT tree, but label with the neural policy's actions
    visited_states = rollout_policy_states(tree_policy, env)
    expert_labels = [neural_policy(s) for s in visited_states]

    # Weight training examples by the neural policy's estimated "criticality" at each state
    weights = [neural_policy.q_value_gap(s) for s in visited_states]   # larger gap = more important to get right

    dataset.extend(zip(visited_states, expert_labels, weights))
    tree_policy = train_weighted_decision_tree(dataset, max_depth=max_depth)
    return tree_policy, dataset

Weighting training examples by how much the neural policy’s Q-values differ between the best and second-best action at each state — a proxy for how “critical” that decision is — focuses the distilled tree’s limited capacity on getting the decisions that matter most right, rather than spreading effort evenly across states where any reasonable action would do about as well. Because the resulting policy is a decision tree, not a neural network, it can in some cases be formally verified against safety properties using standard program-verification techniques that simply don’t apply to a neural network’s continuous weights.

Reward Decomposition: Explaining Decisions in Terms of Objectives

When a task naturally involves multiple sub-objectives (connecting directly to the multi-objective RL techniques covered in a companion article), reward decomposition trains separate Q-value heads for each component reward, so a decision can be explained in terms of which objective was decisive rather than only a single opaque combined number:

class DecomposedQNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, n_reward_components, hidden_dim=256):
        super().__init__()
        self.shared_encoder = nn.Sequential(nn.Linear(state_dim, hidden_dim), nn.ReLU())
        self.component_heads = nn.ModuleList([
            nn.Linear(hidden_dim, action_dim) for _ in range(n_reward_components)
        ])

    def forward(self, state):
        features = self.shared_encoder(state)
        component_qs = [head(features) for head in self.component_heads]   # e.g., [speed_Q, safety_Q, energy_Q]
        return component_qs, sum(component_qs)   # individual components AND their sum

def explain_decision(component_qs, action, component_names):
    return {name: q[action].item() for name, q in zip(component_names, component_qs)}

This lets a human ask “why did the agent brake here” and get back a genuinely informative answer — “because the safety component’s Q-value dominated the speed component’s for this action” — rather than only a single combined score with no way to attribute it to a specific underlying concern.

Comparison Table

Technique	Explains	Output
Saliency Maps	Individual decisions	Which parts of the state input mattered
Policy Distillation (VIPER)	The overall policy logic	An inspectable, potentially verifiable decision tree
Reward Decomposition	Individual decisions, in terms of objectives	Per-objective Q-value breakdown

Applications

Safety-critical deployment validation — verifying a policy doesn’t rely on spurious correlations or fail on plausible edge cases before it’s trusted with real-world consequences.
Regulatory and compliance contexts — domains like healthcare and finance often require some form of explainability before an automated decision system can be deployed at all.
Debugging RL training failures — saliency maps and reward decomposition are both practical tools for diagnosing why a policy converged to unexpected or undesired behavior during development, not just at deployment time.
Building operator trust — in human-in-the-loop systems (robotics supervised by a human operator, clinical decision support), interpretable explanations are often necessary for the human to appropriately calibrate how much to trust the system’s recommendations.

Key Learnings

Different interpretability techniques answer different questions, and picking the right one matters. Saliency maps explain single decisions; policy distillation explains the overall strategy; reward decomposition explains decisions in terms of competing objectives — none of the three substitutes for the others.
Distillation into an interpretable model trades some performance for inspectability, and that trade-off is often worth it specifically in high-stakes settings. VIPER’s decision tree won’t match the original neural network’s performance exactly, but the ability to formally verify or simply read the resulting policy can matter more than a small performance gap in a safety-critical deployment.
Interpretability and the multi-objective framing from a companion article compose naturally. Reward decomposition is essentially “keep the multiple objectives from multi-objective RL separate all the way through to the explanation,” rather than collapsing them into one number the moment training starts.

References

Greydanus, S. et al. (2018). Visualizing and Understanding Atari Agents.
Bastani, O., Pu, Y., Solar-Lezama, A. (2018). Verifiable Reinforcement Learning via Policy Extraction.
Juozapaitis, Z. et al. (2019). Explainable Reinforcement Learning via Reward Decomposition. IJCAI Workshop on Explainable AI.