The Complete Reinforcement Learning Guide
Every article on this site, organized into a curriculum. Start with Value-Based Methods or Policy-Based and Actor-Critic Methods if you're new to RL — everything else builds on those two families.
Value-Based Methods
Learning a value function and acting greedily on it.
From Tabular Q-Learning to Rainbow: A Complete Tour of Value-Based Reinforcement Learning
Every major value-based RL technique explained in order of invention — tabular Q-learning, DQN, Double DQN, Dueling DQN, prioritized replay, multi-step returns, distributional RL, noisy networks — culminating in Rainbow, plus where each technique gets used in practice.
SARSA, Expected SARSA, and TD(λ): The Classical Foundations of Temporal-Difference Control
The on-policy TD control lineage that predates and underlies deep Q-learning — SARSA, Expected SARSA, and TD(λ) with eligibility traces — explained with derivations, code, and where these classical methods still matter.
Dynamic Programming for RL: Policy Iteration and Value Iteration
The planning algorithms that work when the environment's model is fully known — Policy Iteration and Value Iteration — explained with the Bellman equations behind them, code, convergence guarantees, and why almost every RL algorithm on this site is trying to approximate what these methods compute exactly.
Distributional Reinforcement Learning: From C51 to QR-DQN, IQN, and D4PG
The full distributional RL family explained — why modeling the entire distribution of returns beats modeling just their mean, then QR-DQN's quantile regression, IQN's implicit quantile sampling, and D4PG's distributional policy gradients for continuous control — with code and applications.
Policy-Based and Actor-Critic Methods
Optimizing a policy directly, with a critic to lower variance.
Policy Gradient Methods: From REINFORCE to Natural Policy Gradient
The foundational family that actor-critic methods build on — REINFORCE, variance reduction with baselines, and Natural Policy Gradient — explained with the derivations, the code, and where each is used.
Actor-Critic Methods Explained: From Vanilla Actor-Critic to A3C, DDPG, TD3, SAC, and PPO
A complete tour of the actor-critic family in reinforcement learning — how combining a policy network with a value network solves what pure policy gradients and pure value-based methods each struggle with, tracing the lineage from vanilla actor-critic through A2C, A3C, DDPG, TD3, SAC, and PPO, with applications for each.
Model-Based and Planning Methods
Learning or using a model of the environment to plan ahead.
Model-Based Reinforcement Learning: From Dyna-Q to MuZero and Dreamer
Every major model-based RL technique explained in order of invention — Dyna-Q, uncertainty-aware planning with PETS, World Models, MuZero, and Dreamer — covering how learning a model of the environment buys enormous sample efficiency, plus where each technique gets used.
Monte Carlo Tree Search and AlphaZero: Planning by Simulating the Future
How Monte Carlo Tree Search builds a search tree through repeated simulation, and how AlphaZero fused it with deep self-play reinforcement learning to reach superhuman performance in Go, Chess, and Shogi with no human game data at all — with code and applications.
Multi-Agent RL
When more than one learning agent shares the environment.
Learning Without a Reward Function
Offline data, demonstrations, preferences, and self-generated goals.
Offline Reinforcement Learning: From Batch-Constrained Q-Learning to Decision Transformer
Every major offline RL technique explained in order of invention — why naive off-policy Q-learning fails on fixed datasets, then BCQ, CQL, IQL, and the sequence-modeling reframing of Decision Transformer — plus where offline RL gets used when online exploration is unsafe or impossible.
Imitation Learning and Inverse RL: From Behavioral Cloning to GAIL
Every major imitation and inverse RL technique explained in order of invention — Behavioral Cloning, DAgger's interactive correction, Maximum Entropy Inverse RL, and adversarial GAIL — covering how to learn from expert demonstrations instead of a hand-designed reward, plus applications.
Preference-Based RL: Learning From Comparisons Instead of a Reward Function
How agents learn when there's no reward function to write down at all — only humans (or an oracle) comparing pairs of trajectories — covering the original Deep RL from Human Preferences pipeline, active query selection, and how this generalizes the RLHF pipeline used for LLM alignment.
Unsupervised RL and Skill Discovery: DIAYN, Empowerment, and Auto-Curricula
How agents learn useful behavior with no reward function at all — Diversity Is All You Need's mutual-information objective for discovering distinct skills, empowerment-driven exploration, and auto-curriculum methods like POET and PAIRED that co-evolve agents and environments — with code and applications.
Goal-Conditioned RL and Hindsight Experience Replay: Learning From Every Failure
How a single policy learns to reach any goal, not just one fixed objective — Universal Value Function Approximators for goal-conditioned learning, and Hindsight Experience Replay's trick of relabeling failed trajectories as successes for the goal they actually reached — with code and applications.
Exploration, Curricula, and Temporal Abstraction
Deciding what to try next, and how to structure long-horizon tasks.
Exploration in Reinforcement Learning: From Epsilon-Greedy to Random Network Distillation
Every major exploration technique explained in order of invention — epsilon-greedy, Upper Confidence Bound, Thompson Sampling, count-based pseudo-counts, curiosity-driven exploration with ICM, and Random Network Distillation — covering how each answers the explore/exploit trade-off, plus applications.
Curriculum Learning for RL: Reverse Curricula, Self-Paced Learning, and Teacher-Student Frameworks
How to order training experience from easy to hard instead of throwing an agent at the full task difficulty from step one — reverse curriculum generation, self-paced difficulty selection, and teacher-student curriculum frameworks — with code and applications, distinct from the environment-co-evolving auto-curriculum methods covered elsewhere.
Hierarchical Reinforcement Learning: From the Options Framework to Feudal Networks
Every major hierarchical RL technique explained in order of invention — the Options framework, the Option-Critic architecture for learning options end-to-end, Feudal Networks' manager-worker split, and HIRO's off-policy correction — covering how temporal abstraction solves long-horizon, sparse-reward tasks, plus applications.
Semi-Markov Decision Processes: The Formal Foundation for Temporally-Extended Actions
The mathematical formalism underlying every hierarchical RL method — how SMDPs generalize MDPs to actions with variable, non-unit duration, and how SMDP Q-learning updates values across those variable time gaps — with code and applications in hierarchical RL and continuous-time systems.
Generalization and Adaptation
Reusing what's been learned across tasks, environments, and devices.
Meta-Reinforcement Learning: Learning to Learn, From RL² to PEARL
Every major meta-RL technique explained in order of invention — RL² baking adaptation into a recurrent network's activations, MAML's gradient-based fast adaptation, and PEARL's probabilistic task inference for efficient off-policy meta-RL — covering how agents learn to adapt to new tasks in a handful of episodes, plus applications.
Transfer and Generalization in RL: Successor Features, Domain Randomization, and Continual Learning
How RL agents reuse what they've learned instead of retraining from scratch — successor features for fast reward transfer, domain randomization and sim-to-real transfer, and continual RL's fight against catastrophic forgetting — with code and applications.
Federated Reinforcement Learning: Training Across Devices That Can't Share Data
How RL is trained across many decentralized agents or devices without any of them sharing raw trajectories — federated averaging adapted for policy and value networks, and the specific challenges non-iid environments create for RL that supervised federated learning doesn't face — with code and applications.
Safety, Robustness, and Multiple Objectives
Respecting hard limits, worst-case outcomes, and competing goals.
Safe and Constrained Reinforcement Learning: CPO, Lagrangian Methods, and Safe Exploration
How RL is made to respect hard safety limits, not just maximize reward — Constrained Policy Optimization's trust-region-with-constraints approach, Lagrangian relaxation methods, and safety shielding for exploration itself — with code and applications in robotics and industrial control.
Risk-Sensitive and Robust Reinforcement Learning: CVaR Optimization and Adversarial Training
How RL is made to care about worst-case outcomes and hostile environment perturbations, not just average-case return — CVaR-based risk-sensitive policy optimization and Robust Adversarial RL's adversary-in-the-loop training — with code and applications in finance and robust robotics.
Multi-Objective Reinforcement Learning: Pareto Fronts and Reward Scalarization
How RL handles competing objectives that can't be collapsed into one number — scalarization methods that reduce multiple objectives to a single reward, and Pareto-front methods that learn a whole family of trade-off policies at once — with code and applications.
Explainable RL: Making Learned Policies Inspectable
How to see inside a black-box RL policy — saliency maps that show which parts of a state drove a decision, policy distillation into interpretable decision trees (VIPER), and reward decomposition that breaks a single Q-value into interpretable components — with code and applications in safety-critical deployment.
Theory and Evaluation
Formal foundations, simplified settings, and evaluating a policy safely.
Reinforcement Learning Under Partial Observability: POMDPs, Belief States, and DRQN
How RL handles environments where the true state isn't directly visible — the POMDP formalism, exact belief-state tracking, and Deep Recurrent Q-Networks that learn to summarize observation history implicitly — with code and applications in robotics and imperfect-information games.
Multi-Armed and Contextual Bandits: Reinforcement Learning Without State Transitions
Bandits explained as the simplest possible RL problem — no state transitions, just repeated action selection under uncertainty — covering epsilon-greedy, UCB1, Thompson Sampling for the stationary case, and LinUCB for contextual bandits, with code and applications in online recommendation and A/B testing.
Off-Policy Evaluation: Estimating a Policy's Value Without Ever Running It
How to answer 'how good would this new policy be?' using only data collected under a different, old policy — importance sampling, weighted and per-decision variants, and doubly robust estimation that combines importance sampling with a learned value model — with code and applications.
Scaling RL
Training across thousands of actors, gradient-free search, and hyperparameters.
Distributed RL Architectures: IMPALA, Ape-X, R2D2, and SEED RL
How reinforcement learning is scaled across thousands of parallel actors — IMPALA's V-trace correction for off-policy staleness, Ape-X's distributed prioritized replay, R2D2's recurrent extension, and SEED RL's centralized-inference architecture — with code and applications.
Gradient-Free Reinforcement Learning: CMA-ES, Evolution Strategies, and Augmented Random Search
The RL family that skips backpropagation entirely — CMA-ES's covariance-adapted search, OpenAI's scalable Evolution Strategies, Augmented Random Search's surprisingly competitive simplicity, and NEAT's topology-evolving neuroevolution — with code and where gradient-free methods still win.
Population-Based Training: Evolving RL Hyperparameters During Training, Not Before It
How Population-Based Training replaces expensive sequential hyperparameter search with a single training run — a population of agents that periodically exploit better performers and explore perturbed hyperparameters on the fly — with code and its role in large-scale systems like AlphaStar.
Applications
Where these algorithm families get used to solve concrete problems.
Reinforcement Learning for LLM Alignment: From Reward Modeling to PPO-Based RLHF and DPO
Every major technique for aligning LLMs with RL explained in order of invention — reward modeling from human preferences, the full PPO-based RLHF pipeline behind InstructGPT-style models, DPO's single-loss reformulation without an explicit RL loop, and GRPO's critic-free alternative — plus applications.
RL for Combinatorial Optimization: Pointer Networks and Learning to Route
How reinforcement learning tackles NP-hard combinatorial problems like the traveling salesman problem — Pointer Networks' sequence-to-sequence pointing mechanism, REINFORCE-trained construction heuristics, and attention-based routing models — with code and applications in logistics and scheduling.
RL for Scientific Discovery: AlphaTensor, AlphaChip, and AlphaDev
How the MCTS, model-based, and distributed RL techniques covered elsewhere on this site were pointed at open scientific and engineering problems — discovering faster matrix multiplication algorithms, designing computer chip floorplans, and finding better sorting routines — with code and what these results reveal about RL's reach beyond games.