Population-Based Training: Evolving RL Hyperparameters During Training, Not Before It

TL;DR

Every algorithm covered elsewhere on this site — PPO’s clip epsilon, SAC’s temperature, DQN’s learning rate — has hyperparameters that matter a great deal and are expensive to tune via traditional grid or random search, since each candidate setting requires a full, separate training run to evaluate. Population-Based Training (PBT) instead trains a whole population of agents simultaneously, each with its own hyperparameters, and periodically has poorly-performing population members exploit (copy the weights and hyperparameters of a better performer) and then explore (randomly perturb those copied hyperparameters) — turning hyperparameter search into something that happens during training instead of before it, for roughly the cost of one extended training run instead of many independent ones.

The Exploit-and-Explore Cycle

graph LR
    A[Population Trains in Parallel] --> B{Bottom Performer?}
    B -->|yes| C[Exploit: copy a top performer's weights]
    C --> D[Explore: perturb hyperparameters]
    D --> A
    B -->|no| A

    style C fill:#6366f1,color:#fff

Problem Statement

Traditional hyperparameter search (grid search, random search, Bayesian optimization) treats each hyperparameter setting as a separate, complete training run to be evaluated after it finishes — expensive, and wasteful in a specific way: a setting that’s great early in training might be terrible later (a high learning rate that speeds up early progress often needs to decay for stable convergence), but static, pre-training hyperparameter search has no way to express “use this rate for the first quarter, that rate afterward” without a human hand-designing a schedule in advance.

The Exploit-and-Explore Loop

PBT (Jaderberg et al., 2017) trains a population of N agents in parallel, each with its own network weights and its own hyperparameters. Periodically (e.g., every few thousand training steps), each population member checks how it’s doing relative to the rest of the population:

def pbt_step(population, eval_fn, exploit_threshold_percentile=20, perturb_factor=1.2):
    performances = [eval_fn(agent) for agent in population]
    ranked_indices = np.argsort(performances)

    n_bottom = int(len(population) * exploit_threshold_percentile / 100)
    bottom_performers = ranked_indices[:n_bottom]
    top_performers = ranked_indices[-n_bottom:]

    for idx in bottom_performers:
        # EXPLOIT: copy weights and hyperparameters from a randomly chosen top performer
        source_idx = np.random.choice(top_performers)
        population[idx].weights = copy.deepcopy(population[source_idx].weights)
        population[idx].hyperparameters = copy.deepcopy(population[source_idx].hyperparameters)

        # EXPLORE: perturb the copied hyperparameters so the population keeps searching
        for key in population[idx].hyperparameters:
            if random.random() < 0.5:
                population[idx].hyperparameters[key] *= perturb_factor
            else:
                population[idx].hyperparameters[key] /= perturb_factor

    return population

Because weights are copied along with hyperparameters (not just hyperparameters alone), a bottom performer doesn’t restart from scratch when it exploits a better agent — it inherits that agent’s partially-trained network too, so the population as a whole makes continuous progress even as individual hyperparameter settings are being discarded and replaced throughout training.

Why This Beats Static Hyperparameter Schedules

The exploit-and-explore cycle implicitly discovers a hyperparameter schedule — since the perturbation-and-selection process happens continuously throughout training, whatever hyperparameter values work best early on tend to spread through the population early, and if different values work better later, the same mechanism finds and propagates those too, all without a human having to specify in advance when or how hyperparameters should change over the course of training.

def pbt_training_loop(population, env, n_generations, steps_per_generation):
    for generation in range(n_generations):
        for agent in population:
            train_for_steps(agent, env, steps_per_generation)   # ordinary RL training, e.g. PPO
        population = pbt_step(population, eval_fn=lambda a: evaluate(a, env))
    return max(population, key=lambda a: evaluate(a, env))

Comparison Table

Applications

• Large-scale competitive game AI training — PBT was a key component of DeepMind’s AlphaStar training pipeline, both for hyperparameter tuning and, combined with the self-play ideas covered in the companion multi-agent RL article, for maintaining a diverse league of strategies.
• Any hyperparameter-sensitive algorithm — PPO’s clip range, SAC’s entropy temperature, and learning rates across nearly every method on this site are all standard PBT targets.
• Production RL training pipelines — where compute for many independent full training runs isn’t available, but running one larger, population-parallel job is.
• Joint architecture and hyperparameter search — PBT’s exploit/explore mechanism generalizes beyond scalar hyperparameters to discrete architectural choices as well, by including them in what gets copied and perturbed.

Key Learnings

1. PBT’s core insight is that hyperparameter search and training don’t have to be separate phases. Doing both simultaneously, with weights carried along through exploitation, avoids the fundamental waste of traditional search — restarting from scratch for every new hyperparameter candidate.
2. The implicit schedule discovery is the real advantage over a fixed hyperparameter setting. A single best static value, found by any traditional search method, is still a compromise across the entire training run; PBT can effectively use different values at different training stages without anyone specifying the schedule.
3. This is a meta-level technique that composes with everything else on this site. PBT doesn’t replace PPO, SAC, or any other algorithm — it wraps around whichever one you’re using, managing its hyperparameters over the course of training rather than proposing a new way to update the policy itself.

References

• Jaderberg, M. et al. (2017). Population Based Training of Neural Networks.
• Vinyals, O. et al. (2019). Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575.