Unsupervised RL and Skill Discovery: DIAYN, Empowerment, and Auto-Curricula

TL;DR

Every method covered elsewhere on this site assumes a reward function is given. Unsupervised RL asks: what can an agent learn with no extrinsic reward at all? DIAYN (Diversity Is All You Need) discovers a whole repertoire of distinct, reusable skills purely by maximizing how distinguishable each skill’s behavior is from the others. Empowerment-driven methods reward an agent for reaching states from which it has maximal future influence over its environment. Auto-curriculum methods like POET and PAIRED go a level further, co-evolving the environment itself alongside the agent to generate an endless, appropriately-difficulty-scaled sequence of new challenges. Used for pre-training reusable skills before task-specific fine-tuning, procedural content generation, and open-ended learning research.

The DIAYN Loop

graph LR
    A[Sample Random Skill z] --> B[Policy Acts Conditioned on z]
    B --> C[Discriminator Guesses Which Skill Produced This State]
    C -->|reward = correct-guess likelihood| B

    style C fill:#6366f1,color:#fff

Problem Statement

Reward functions are often unavailable, expensive to specify correctly, or actively distort behavior when hand-designed poorly (reward hacking). Separately, even when a real task reward exists, an agent that already has a broad, general-purpose repertoire of skills to draw on can learn that task far faster than one starting from nothing. Unsupervised RL’s goal is to produce that broad repertoire, or at least productive behavior, without needing a task-specific reward signal to bootstrap from.

DIAYN: Discovering Skills by Maximizing Their Distinguishability

DIAYN (Eysenbach et al., 2018) frames “useful skill” information-theoretically: train a policy conditioned on a latent skill variable z, and reward it not for accomplishing any task, but for visiting states from which a separately trained discriminator can correctly infer which skill z produced that state. This is a mutual-information objective — maximize I(states; skill) — implemented via a straightforward classification reward:

def diayn_reward(discriminator, state, skill_z, prior_prob_z):
    predicted_skill_probs = discriminator(state)   # discriminator's belief over which skill this state came from
    log_prob_correct_skill = torch.log(predicted_skill_probs[skill_z] + 1e-8)
    return log_prob_correct_skill - torch.log(prior_prob_z + 1e-8)   # pseudo-reward, no task reward involved

def diayn_training_step(policy, discriminator, env, n_skills):
    skill_z = np.random.randint(n_skills)          # sample a random skill to practice this episode
    state = env.reset()
    trajectory = []
    while not env.done:
        action = policy(state, skill_z)
        next_state = env.step(action)
        reward = diayn_reward(discriminator, next_state, skill_z, prior_prob_z=1.0 / n_skills)
        trajectory.append((state, action, reward))
        state = next_state
    return trajectory

The discriminator and policy train against each other: the policy is pushed to make each skill visit distinctive states (so the discriminator can tell skills apart), and the discriminator gets better at telling them apart as the policy differentiates further — the emergent result, with no task reward anywhere in the loop, is a set of skills that are behaviorally diverse (e.g., in a legged robot, different skills often correspond to distinct gaits or movement directions) purely because diversity is what the objective rewards.

Empowerment: Rewarding Influence Over the Future

A related but distinct intrinsic objective, empowerment, rewards an agent for reaching states from which its actions have maximal predictable influence over future outcomes — formally, the channel capacity between an agent’s actions and future states, I(actions; future states).

def empowerment_bonus(forward_model, state, action_sequence_samples):
    # Approximate how much control the agent has: how distinguishable are the outcomes
    # of different action sequences from this state?
    predicted_outcomes = [forward_model(state, actions) for actions in action_sequence_samples]
    outcome_diversity = estimate_mutual_information(action_sequence_samples, predicted_outcomes)
    return outcome_diversity

Intuitively, empowerment rewards an agent for staying in “powerful” positions — a robot standing upright (many possible next moves) scores higher than one that has fallen over (few possible next moves) — without ever needing a task reward to say that standing is good.

Auto-Curricula: Co-Evolving the Environment With the Agent

DIAYN and empowerment generate intrinsic reward within a fixed environment. POET (Paired Open-Ended Trailblazer, Wang et al., 2019) and PAIRED (Dennis et al., 2020) instead treat the environment’s difficulty and structure as something to evolve alongside the agent, generating a never-ending, appropriately-paced curriculum automatically:

def poet_style_loop(population_of_env_agent_pairs, mutation_rate=0.1):
    for env, agent in population_of_env_agent_pairs:
        agent.train_on(env)                              # normal RL training

    for env, agent in population_of_env_agent_pairs:
        if agent.performance_on(env) > MASTERY_THRESHOLD:
            new_env = env.mutate(mutation_rate)           # environment gets harder/different
            if is_appropriately_challenging(new_env, agent):
                population_of_env_agent_pairs.append((new_env, agent.clone()))

    periodically_transfer_agents_across_environments(population_of_env_agent_pairs)

PAIRED frames this as a three-player game specifically: an environment-generating adversary is rewarded for creating levels that a “protagonist” agent fails at but an “antagonist” agent (a stronger reference policy) succeeds at — targeting a curriculum of levels that are hard but not literally unsolvable, which turns out to be a more effective and stable auto-curriculum signal than simply maximizing an agent’s regret or failure rate directly.

Comparison Table

Applications

• Pre-training reusable skill repertoires — DIAYN-style skill discovery before fine-tuning on a specific downstream task can dramatically speed up learning versus starting from a randomly initialized policy.
• Procedural content generation — auto-curriculum methods like POET have direct applications in automatically generating game levels or simulation scenarios matched to a player or agent’s current skill level.
• Robustness testing and open-ended learning research — PAIRED-style adversarial environment generation surfaces edge cases and failure modes a fixed, hand-designed curriculum might never expose an agent to.
• Sparse or absent reward domains — any setting where specifying a good task reward is difficult but where “behave in an interesting, diverse, or influential way” is a reasonable stand-in objective.

Key Learnings

1. Information-theoretic objectives (mutual information between behavior and some target variable) are a surprisingly general recipe for producing useful behavior without a task reward. Both DIAYN (states/skill) and empowerment (actions/future states) are instances of the same underlying pattern.
2. A good auto-curriculum has to target the edge of an agent’s current ability, not just difficulty in the abstract. PAIRED’s three-player framing exists specifically to avoid generating environments that are either trivially easy or hopelessly unsolvable — both are equally useless for learning.
3. Unsupervised RL and standard RL are complementary stages, not competitors. The typical intended use is unsupervised pre-training to build general competence or a skill repertoire, followed by ordinary reward-driven fine-tuning (any method elsewhere on this site) once a real task and reward are available.

References

• Klyubin, A., Polani, D., Nehaniv, C. (2005). Empowerment: A Universal Agent-Centric Measure of Control. IEEE Congress on Evolutionary Computation.
• Eysenbach, B. et al. (2018). Diversity is All You Need: Learning Skills without a Reward Function.
• Wang, R. et al. (2019). Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions.
• Dennis, M. et al. (2020). Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design.