Curriculum Learning for RL: Reverse Curricula, Self-Paced Learning, and Teacher-Student Frameworks

TL;DR

Throwing an agent at a task’s full difficulty from the very first training step often wastes enormous amounts of exploration on a problem that’s simply too hard to get any reward signal from yet. Curriculum learning orders training experience from easy to hard instead. Reverse curriculum generation starts an agent very close to the goal (where success is nearly guaranteed) and gradually expands the starting-state distribution outward as competence grows. Self-paced learning lets the agent’s own performance determine how quickly difficulty ramps up, rather than a fixed hand-designed schedule. Teacher-student frameworks use a separate “teacher” policy or process to choose what task or difficulty level the “student” trains on next. This is distinct from the auto-curriculum methods (POET, PAIRED) covered in the companion unsupervised RL article, which co-evolve the environment itself — curriculum learning here is about sequencing existing tasks or start states, not generating new environments.

Three Ways to Sequence Difficulty

graph TD
    A[Order Training Experience Easy to Hard] --> B[Reverse Curriculum: expand outward from the goal]
    A --> C[Self-Paced: agent's own performance sets the pace]
    A --> D[Teacher-Student: a separate process picks the next task]

    style C fill:#6366f1,color:#fff

Problem Statement

Sparse-reward, long-horizon tasks are exactly where random exploration is least likely to stumble onto success early in training — a robot arm starting from a random position, tasked with placing an object in a specific target far away, may need an implausibly long, precisely-correct sequence of actions before ever seeing its first reward. Curriculum learning’s core bet is that the same task, approached with an easier version first, produces a policy that can then generalize (or be gradually adapted) to the full difficulty, in far less total training time than attacking the hard version directly.

Reverse Curriculum Generation: Start Near the Goal

Reverse curriculum generation (Florensa et al., 2017) starts training episodes very close to the goal state, where the agent can succeed almost immediately by chance, and gradually samples starting states further and further from the goal as the agent’s success rate at the current distance improves:

def reverse_curriculum_step(agent, goal_state, start_state_buffer, env, success_threshold=0.8):
    # Sample starting states near states the agent already succeeds from
    candidate_starts = sample_nearby_states(start_state_buffer, perturbation_std=0.1)

    success_rates = {}
    for start in candidate_starts:
        successes = [rollout_and_check_success(agent, env, start, goal_state) for _ in range(10)]
        success_rates[start] = np.mean(successes)

    # Keep starts that are hard but not impossible: neither trivial nor unsolved
    good_starts = [s for s, rate in success_rates.items() if 0.1 < rate < success_threshold]
    start_state_buffer.extend(good_starts)   # these become next round's "near" states, expanding the frontier

    return start_state_buffer

The frontier of starting states expands outward at roughly the pace the agent can actually handle — states that are already easy (success rate near 1) stop being useful to add, and states that are still essentially unsolved (success rate near 0) are correctly excluded until closer states have been mastered first, extending the frontier naturally.

Self-Paced Learning: The Agent Sets Its Own Difficulty

Rather than an external process choosing the curriculum, self-paced learning lets the current policy’s own performance directly determine task difficulty, typically by maintaining a continuous difficulty parameter and only advancing it once performance at the current level clears a threshold:

def self_paced_difficulty_update(current_difficulty, recent_success_rate,
                                   advance_threshold=0.75, retreat_threshold=0.3, step_size=0.05):
    if recent_success_rate > advance_threshold:
        return min(1.0, current_difficulty + step_size)   # ready for harder tasks
    elif recent_success_rate < retreat_threshold:
        return max(0.0, current_difficulty - step_size)   # struggling, ease back off
    return current_difficulty   # in a good zone, stay here

This creates a moving target that tracks the agent’s actual competence rather than a fixed, pre-planned schedule — if the agent plateaus or regresses (e.g., due to catastrophic forgetting while learning something else, connecting to the continual RL problem covered in the companion transfer learning article), the curriculum backs off automatically rather than continuing to push a difficulty level the agent can no longer handle.

Teacher-Student Curriculum Frameworks

A teacher — which can be a simple heuristic, a separately trained policy, or even a bandit algorithm — explicitly selects which task or environment configuration the student trains on next, typically optimizing for tasks where the student is currently making the fastest learning progress rather than tasks that are simply easy or simply hard:

def teacher_select_task(task_pool, student_progress_history, exploration_bonus=0.1):
    # Estimate learning progress per task: how much has the student's performance improved recently?
    learning_progress = {
        task: recent_improvement_rate(student_progress_history[task]) for task in task_pool
    }

    # Bandit-style selection favoring tasks with high progress, with exploration for under-tried tasks
    scores = {
        task: progress + exploration_bonus / np.sqrt(1 + len(student_progress_history[task]))
        for task, progress in learning_progress.items()
    }
    return max(scores, key=scores.get)

Framing task selection as a bandit problem over “which task yields the most learning progress right now” (rather than “which task is easiest” or a fixed schedule) is what connects this family directly back to the multi-armed bandit techniques covered in a companion article — the teacher is running its own exploration/exploitation problem, just over curriculum choices instead of environment actions.

Comparison Table

Applications

• Sparse-reward robotic manipulation — reverse curriculum generation is particularly well suited to goal-reaching tasks, complementing the goal-conditioned RL and HER techniques covered in a companion article.
• Multi-task and lifelong learning systems — teacher-student frameworks naturally extend to selecting which of many available tasks to practice next, connecting to the continual RL problem of not forgetting earlier tasks.
• Sim-to-real robotics training — gradually increasing simulation difficulty or domain randomization range as competence grows is a curriculum-learning-flavored complement to the domain randomization technique covered in the companion transfer learning article.
• Any long-horizon, sparse-reward task — general applicability wherever training on the full task difficulty from the start produces too little learning signal to make progress.

Key Learnings

1. Curriculum learning here is about sequencing existing difficulty, not generating new environments. This is the key distinction from the auto-curriculum methods (POET, PAIRED) in the companion unsupervised RL article — those co-evolve the environment itself; these techniques order training experience within a task that’s already defined.
2. The right difficulty level is a moving target, not a fixed schedule. Self-paced learning’s advance/retreat mechanism and reverse curriculum’s frontier expansion both exist because a pre-planned, fixed-in-advance schedule can’t adapt to an agent that’s learning faster or slower than expected.
3. “Maximize learning progress,” not “minimize difficulty” or “maximize immediate reward,” is often the right objective for a curriculum. Teacher-student frameworks make this explicit by directly optimizing for tasks where the student is currently improving fastest — a subtly different and often more effective target than simply ordering tasks from easy to hard.

References

• Bengio, Y. et al. (2009). Curriculum Learning. ICML.
• Florensa, C. et al. (2017). Reverse Curriculum Generation for Reinforcement Learning.
• Matiisen, T. et al. (2017). Teacher-Student Curriculum Learning.