Imitation Learning and Inverse RL: From Behavioral Cloning to GAIL

TL;DR

Sometimes you don’t have a reward function at all — you only have examples of an expert doing the task well. Behavioral Cloning treats this as ordinary supervised learning, but small errors compound over time into states the expert never showed it (the compounding-error problem); DAgger fixes that by interactively querying the expert on the learner’s own visited states; Inverse RL goes further and tries to recover the reward function the expert must have been optimizing, rather than just their actions; and GAIL reframes that as an adversarial game, learning to imitate without ever solving for an explicit reward. Used in autonomous driving from human driving logs, robotic manipulation from teleoperation demonstrations, and bootstrapping game AI before RL fine-tuning.

Problem Statement

Designing a good reward function is often harder than it looks — a poorly shaped reward leads to reward hacking, where the agent optimizes the letter of the reward instead of the intended behavior. In many real domains (driving, surgery, dexterous manipulation), it’s far easier to show an agent good behavior than to write down a reward function that provably produces it. Imitation learning and inverse RL are two different answers to “how do you learn from demonstrations instead of a reward signal?”

Behavioral Cloning: Supervised Learning on Expert Trajectories

The simplest approach: treat state-action pairs from expert demonstrations as a labeled dataset and train a policy with ordinary supervised learning.

def behavioral_cloning_loss(policy, expert_states, expert_actions):
    predicted_actions = policy(expert_states)
    return F.mse_loss(predicted_actions, expert_actions)   # or cross-entropy for discrete actions

This works well as long as the learner stays close to states the expert actually demonstrated. The moment the learner drifts even slightly off the expert’s distribution — due to a small prediction error — it finds itself in a state it was never trained on, makes another error, and the errors compound: this is the well-known problem that makes naive BC fragile on long-horizon tasks.

DAgger: Interactive Expert Correction

DAgger (Dataset Aggregation, Ross et al., 2011) fixes the compounding-error problem directly: instead of training once on a fixed expert dataset, it repeatedly rolls out the current learner in the environment, has the expert label what it should have done at every state the learner actually visited, and adds those corrections back into the training set.

def dagger_iteration(policy, expert, env, dataset):
    trajectory_states = rollout_policy_states(policy, env)      # learner's own visited states
    expert_labels = [expert.query(s) for s in trajectory_states] # expert corrects them

    dataset.extend(zip(trajectory_states, expert_labels))
    policy = train_supervised(policy, dataset)                   # retrain on the aggregated set
    return policy, dataset

Because the training distribution now includes states the learner actually drifts into (not just states the expert visited), the compounding-error problem is addressed at its source rather than patched after the fact.

Inverse Reinforcement Learning: Recovering the Reward

BC and DAgger both copy the expert’s actions directly. Inverse RL asks a different question: what reward function would explain the expert’s behavior as (near-)optimal? Recovering an explicit reward, rather than just actions, has an advantage — a learned reward can generalize to states and even related tasks the expert demonstrations never covered. Maximum Entropy IRL (Ziebart et al., 2008) formalizes this by finding the reward function under which the expert’s demonstrated trajectories are exponentially more likely than alternatives, while otherwise assuming maximum uncertainty (max entropy) about everything else — avoiding the ambiguity that many different reward functions could explain the same finite set of demonstrations.

def maxent_irl_gradient(reward_net, expert_trajectories, policy_trajectories, feature_fn):
    expert_feature_expectation = feature_fn(expert_trajectories).mean(dim=0)
    policy_feature_expectation = feature_fn(policy_trajectories).mean(dim=0)

    # Gradient pushes reward to favor expert-visited features over policy-visited ones
    return expert_feature_expectation - policy_feature_expectation

GAIL: Imitation as an Adversarial Game

Classic IRL requires solving a full RL problem inside every iteration of the reward-learning loop, which is expensive. GAIL (Generative Adversarial Imitation Learning, Ho & Ermon, 2016) reframes imitation as a GAN-style game instead: a discriminator learns to distinguish expert trajectories from the learner’s trajectories, and the learner’s policy is trained (via ordinary policy gradient/actor-critic methods) to fool the discriminator — using the discriminator’s confusion as an implicit reward signal.

def gail_discriminator_loss(discriminator, expert_batch, policy_batch):
    expert_preds = discriminator(expert_batch)
    policy_preds = discriminator(policy_batch)
    return -(torch.log(expert_preds + 1e-8).mean() + torch.log(1 - policy_preds + 1e-8).mean())

def gail_policy_reward(discriminator, state_action):
    # Reward the policy for producing (s, a) pairs the discriminator can't tell from expert data
    return -torch.log(1 - discriminator(state_action) + 1e-8)

This never explicitly recovers a reward function the way classic IRL does — the discriminator implicitly encodes it — but it’s dramatically cheaper to train and produces policies that match expert behavior distributionally, not just at the specific states the expert demonstrated.

Comparison Table

Applications

• Autonomous driving — learning steering and control policies directly from large logged datasets of human driving.
• Robotic manipulation from demonstration — teleoperated demonstrations of grasping and assembly tasks are far cheaper to collect than hand-engineering a dense reward for dexterous manipulation.
• Game AI bootstrapping — initializing a policy via behavioral cloning on human replay data before fine-tuning further with RL, which is often faster than learning from scratch with pure exploration.
• Reward design for downstream RL — IRL-recovered rewards can be reused to train new agents with standard RL once the reward itself is known.

Key Learnings

1. The compounding-error problem is the central failure mode of naive imitation learning, and DAgger’s fix — training on the learner’s own visited states, not just the expert’s — is a pattern that reappears across robust imitation learning research.
2. Copying actions and recovering rewards are different goals with different generalization properties. BC/DAgger produce a policy that mimics the expert; IRL produces a reward that can, in principle, explain and generalize the expert’s intent to new situations.
3. GAIL’s adversarial framing made imitation learning dramatically more practical by replacing IRL’s expensive inner-loop RL solve with a single end-to-end adversarial training process, at the cost of no longer having an explicit, reusable reward function at the end.