Transfer and Generalization in RL: Successor Features, Domain Randomization, and Continual Learning

TL;DR

Every method covered elsewhere on this site, by default, learns one policy for one fixed task and environment, from scratch. Transfer and generalization techniques attack that limitation from three different angles: successor features decompose value into a reusable “how the world works” part and a swappable “what I currently care about” part, enabling near-instant transfer when only the reward changes; domain randomization trains across many randomized simulated variations of an environment so the resulting policy generalizes to the real world without ever seeing it during training; and continual RL confronts what happens when an agent must learn a sequence of tasks over time without catastrophically forgetting earlier ones. Used in robotics sim-to-real transfer, multi-task and lifelong learning systems, and any setting where retraining from scratch per task is too expensive.

Three Angles on Reuse

graph TD
    A[Don't Retrain From Scratch] --> B[Successor Features: reuse dynamics, swap reward]
    A --> C[Domain Randomization: train across simulated variation]
    A --> D[Continual RL: don't forget earlier tasks]

    style B fill:#6366f1,color:#fff

Problem Statement

Training a policy from scratch for every new task or environment throws away everything learned about tasks that came before, even when much of that experience — how physics works, how obstacles behave, general world dynamics — is directly reusable. Separately, a policy trained entirely in simulation typically fails when deployed on the real system it was meant to control, because simulators are never perfectly accurate models of reality. These are related but distinct generalization failures, and each has its own family of fixes.

Successor Features: Separating Dynamics From Reward

Successor Features (Barreto et al., 2017, building on Dayan’s 1993 Successor Representation) decompose the Q-function into two pieces: a successor feature vector ψ(s,a) capturing expected future feature occupancy (independent of any specific reward), and a reward weight vector w describing what the current task cares about. The Q-function is just their dot product:

def q_from_successor_features(successor_features_psi, reward_weights_w):
    return successor_features_psi @ reward_weights_w   # Q(s,a) = psi(s,a) . w

def transfer_to_new_task(psi_network, state, action, new_reward_weights):
    # No retraining of psi needed — only the reward weights change for a new task
    psi = psi_network(state, action)
    return psi @ new_reward_weights

If a new task only changes what’s rewarded (not the environment’s underlying dynamics), transferring to it can be nearly instantaneous: ψ — trained once, capturing how the world behaves — never needs retraining, and the agent only needs to estimate the new w, often from a handful of reward samples, then combine it with the already-learned ψ to get a good Q-function immediately.

Domain Randomization: Training Across Simulated Variation

Domain randomization (Tobin et al., 2017) addresses the sim-to-real gap directly: rather than training in one fixed, “accurate” simulator, randomize simulation parameters — friction, mass, visual textures, lighting, sensor noise — widely across training episodes, so the real world simply looks like “one more randomized variation” the policy has already learned to handle.

def domain_randomized_episode(env, randomization_ranges):
    randomized_params = {
        param: np.random.uniform(low, high)
        for param, (low, high) in randomization_ranges.items()
    }
    env.set_simulation_parameters(randomized_params)   # friction, mass, textures, etc.
    return env.reset()

The core bet is that a policy robust to a wide enough distribution of simulated conditions will treat the specific, unknown real-world conditions as just another sample from that distribution, rather than as a fundamentally out-of-distribution shock the way a policy trained on one narrow, fixed simulation setting would.

Continual RL: Learning a Sequence of Tasks Without Forgetting

Continual (lifelong) RL confronts a different problem: when an agent trains on a sequence of tasks one after another, ordinary gradient-based training on a new task tends to overwrite what was learned for earlier ones — catastrophic forgetting. Elastic Weight Consolidation (Kirkpatrick et al., 2017), adapted from supervised continual learning to RL, addresses this by penalizing changes to parameters that were important for earlier tasks, weighted by an estimate of how much each parameter mattered:

def ewc_loss(current_params, old_params, fisher_importance, new_task_loss, ewc_lambda=100):
    # Penalize drifting parameters that were important (high Fisher importance) for past tasks
    consolidation_penalty = sum(
        fisher_importance[name] * (current_params[name] - old_params[name]).pow(2).sum()
        for name in current_params
    )
    return new_task_loss + ewc_lambda * consolidation_penalty

The Fisher importance estimate is computed after finishing each task, capturing which parameters that task’s performance was most sensitive to — those get anchored in place during subsequent tasks, while parameters the earlier task didn’t rely on remain free to adapt to new ones.

Comparison Table

Applications

• Robotics sim-to-real transfer — domain randomization is close to a standard practice for any policy trained in simulation and deployed on physical hardware, from manipulation to locomotion.
• Multi-task and goal-conditioned systems — successor features are particularly well suited to settings with a fixed environment but a changing reward or goal, such as a household robot serving different user requests.
• Lifelong learning systems — any agent expected to keep learning new tasks over an extended deployment lifetime without a full retrain needs some form of continual learning safeguard against forgetting.
• Rapid task adaptation — successor features’ near-instant reward transfer is a lighter-weight alternative to the meta-RL techniques (RL², MAML, PEARL) covered in a companion article, specifically when only the reward, not the dynamics, changes between tasks.

Key Learnings

1. Not every generalization problem is the same problem, despite looking similar on the surface. Successor features solve “the reward changed,” domain randomization solves “training and deployment environments differ,” and continual RL solves “I have to learn tasks in sequence without forgetting” — conflating them leads to reaching for the wrong tool.
2. Domain randomization’s central bet is that breadth of training variation beats accuracy of any single simulation. A wide but imperfect distribution of simulated conditions often transfers better than one very accurate but narrow simulator, because the real world is treated as in-distribution rather than a shock.
3. Catastrophic forgetting is a direct consequence of how gradient descent works, not a bug specific to RL. EWC’s fix — explicitly penalizing drift in parameters that mattered before — is borrowed from supervised continual learning, a reminder that RL-specific problems sometimes have non-RL-specific solutions.

References

• Dayan, P. (1993). Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Computation, 5(4).
• Barreto, A. et al. (2017). Successor Features for Transfer in Reinforcement Learning.
• Tobin, J. et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.
• Kirkpatrick, J. et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks.