Meta-Reinforcement Learning: Learning to Learn, From RL² to PEARL

TL;DR

Ordinary RL trains one policy for one task from scratch, which is wasteful when you actually care about a distribution of related tasks and want fast adaptation to new ones. Meta-RL trains across many tasks so the agent learns an efficient adaptation strategy itself — RL² bakes that strategy into the hidden state of a recurrent network trained across episode sequences with no explicit adaptation step at all; MAML instead learns an initialization from which a few ordinary gradient steps adapt quickly to a new task; and PEARL learns a probabilistic embedding of “what task am I probably in” from a handful of transitions, enabling efficient off-policy meta-training. Used in robotics that must adapt quickly to new environments or damaged hardware, personalization, and sim-to-real transfer.

Problem Statement

A robot trained to walk on flat ground and then deployed on gravel has to either fail or retrain from scratch with ordinary RL — but a human or animal adapts to a new surface within a few strides. Meta-RL (“learning to learn”) trains an agent across a distribution of related tasks during training, with the explicit goal that when a new, related task shows up at test time, the agent adapts in a handful of episodes instead of a full training run.

RL²: Adaptation as a Side Effect of Recurrence

RL² (Duan et al., 2016; also proposed concurrently as “Learning to Reinforcement Learn”) takes a strikingly simple approach: train an ordinary recurrent policy (an RNN/LSTM) not on single episodes, but on sequences of episodes drawn from different tasks, with the hidden state never reset between episodes within a sequence, and reward/action fed back as input at every step.

def rl2_rollout(policy_rnn, task, n_episodes_per_task=5):
    hidden_state = policy_rnn.init_hidden()
    trajectory = []

    for episode in range(n_episodes_per_task):
        state = task.reset()
        done = False
        prev_action, prev_reward = None, 0.0
        while not done:
            # Previous action/reward fed back in — the RNN's hidden state IS the adaptation
            action, hidden_state = policy_rnn(state, prev_action, prev_reward, hidden_state)
            state, reward, done = task.step(action)
            prev_action, prev_reward = action, reward
            trajectory.append((state, action, reward))

    return trajectory

There is no separate “adaptation phase” — the RNN’s hidden state accumulates task-relevant information across the first episode or two purely as a side effect of ordinary training, and by later episodes in the sequence, the same fixed network is behaving like it has already adapted to the specific task, without a single explicit gradient update at test time.

MAML: Learning an Initialization That Adapts Fast

MAML (Model-Agnostic Meta-Learning, Finn et al., 2017) takes a different, model-agnostic approach: rather than relying on recurrence, it explicitly optimizes the policy’s initial parameters so that a small number of ordinary gradient steps on a new task’s data produces a good task-specific policy.

def maml_meta_update(policy, task_batch, inner_lr=0.01, outer_lr=0.001):
    meta_gradients = []
    for task in task_batch:
        adapted_params = policy.parameters().clone()

        # Inner loop: a few ordinary gradient steps, specific to this task
        support_data = task.sample_trajectories(policy)
        inner_loss = compute_policy_loss(policy, support_data, adapted_params)
        adapted_params = adapted_params - inner_lr * grad(inner_loss, adapted_params)

        # Outer loop: evaluate the ADAPTED policy on fresh data, gradient flows back to the ORIGINAL init
        query_data = task.sample_trajectories(policy, params=adapted_params)
        outer_loss = compute_policy_loss(policy, query_data, adapted_params)
        meta_gradients.append(grad(outer_loss, policy.parameters()))

    policy.parameters() -= outer_lr * average(meta_gradients)

The key trick is differentiating through the inner-loop adaptation step — the meta-gradient answers “how should I change my initialization so that a few gradient steps from here lands somewhere good,” which is a fundamentally different mechanism from RL²’s recurrence-based adaptation, though both target the same goal.

PEARL: Probabilistic Task Inference for Off-Policy Efficiency

Both RL² and vanilla MAML are typically trained on-policy, which is sample-inefficient — a real limitation for meta-RL, where the whole point is often to reduce the amount of data needed per task. PEARL (Rakelly et al., 2019) separates the problem into two explicit pieces: a probabilistic context encoder that infers a latent variable representing “which task am I probably in” from a handful of recent transitions, and an otherwise-ordinary off-policy actor-critic (like SAC, covered in the companion actor-critic article) conditioned on that inferred latent.

def pearl_task_inference(context_encoder, recent_transitions):
    # Infers a posterior distribution over the latent task variable z
    context_embedding = context_encoder(recent_transitions)
    z_mean, z_std = context_embedding.chunk(2, dim=-1)
    z = z_mean + z_std * torch.randn_like(z_std)   # sample from the posterior
    return z

def pearl_policy_step(sac_policy, state, task_latent_z):
    return sac_policy(torch.cat([state, task_latent_z], dim=-1))

Because the underlying policy learner is an ordinary off-policy actor-critic, PEARL can reuse a replay buffer across meta-training the way SAC or TD3 does, which is what gives it a large sample-efficiency edge over on-policy meta-RL methods like RL² and standard MAML.

Comparison Table

Applications

• Robotics that must adapt to new conditions — new terrain, payloads, or partial hardware damage, where a few episodes of adaptation is acceptable but full retraining is not.
• Sim-to-real transfer — training across a distribution of simulated task variations so the resulting policy adapts quickly to the specific real-world instance it’s deployed on.
• Personalization — treating each user as a distinct “task” and meta-training a base policy that adapts quickly to individual user preferences from limited interaction data.

Key Learnings

1. “Learning to learn” is really about where the adaptation mechanism lives. RL² puts it in a recurrent hidden state, MAML puts it in a learned initialization plus ordinary gradients, and PEARL puts it in an explicit probabilistic task-inference module — three structurally different answers to the same goal.
2. Sample efficiency during meta-training is a real constraint, not an afterthought. PEARL’s entire design is a response to the fact that on-policy methods like RL² and vanilla MAML need far more data per task than off-policy methods do, which matters enormously when “task” itself is an expensive-to-sample dimension.
3. A good meta-RL agent doesn’t need to be told it’s adapting. In all three methods, the mechanism that makes fast adaptation possible is invisible from the outside — there’s no explicit “detect new task, now adapt” switch, which is part of what makes meta-RL a genuinely different paradigm from fine-tuning a single-task policy.