TL;DR
Model-free RL (Q-learning, actor-critic) learns purely from real environment interaction, which is sample-hungry. Model-based RL instead learns a model of the environment’s dynamics and uses it to plan or generate synthetic experience — starting with tabular Dyna-Q interleaving real and simulated updates, through uncertainty-aware PETS planning, World Models that hallucinate entire training rollouts, MuZero which plans with Monte Carlo Tree Search over a learned model without ever being told the environment’s rules, and Dreamer which learns the policy entirely inside a compact latent “imagination.” Used in board games (AlphaZero/MuZero), robotics where real-world samples are expensive, and any domain where a simulator is either unavailable or too slow.
Problem Statement
A model-free agent needs to actually experience an outcome to learn from it. In robotics or any physical system, real-world rollouts are slow, expensive, or risky — you can’t run millions of Atari-speed episodes on a real robot arm. Model-based RL asks a different question: what if the agent also learns p(s'|s,a) and r(s,a), a predictive model of the world, and uses that model to plan ahead or generate extra training data for free?
Dyna-Q: Interleaving Real and Simulated Learning
Dyna-Q (Sutton, 1990) is the simplest possible version of the idea: learn a tabular model from real transitions, then after every real step, also run several extra Q-learning updates against simulated transitions sampled from that model.
def dyna_q_step(Q, model, s, a, r, s_next, planning_steps=10, alpha=0.1, gamma=0.99):
Q[s][a] += alpha * (r + gamma * max(Q[s_next].values()) - Q[s][a]) # real update
model[(s, a)] = (r, s_next) # update learned model
for _ in range(planning_steps): # simulated updates
s_sim, a_sim = random.choice(list(model.keys()))
r_sim, s_next_sim = model[(s_sim, a_sim)]
Q[s_sim][a_sim] += alpha * (r_sim + gamma * max(Q[s_next_sim].values()) - Q[s_sim][a_sim])
Every real transition is worth 1 + planning_steps updates instead of one, which is the entire sample-efficiency argument for model-based RL in a single algorithm.
PETS: Planning Under Model Uncertainty
Neural network dynamics models are wrong, especially early in training and far from the data they’ve seen — planning against a single point-estimate model lets the planner exploit the model’s mistakes. PETS (Chua et al., 2018) instead learns an ensemble of probabilistic dynamics models, and plans (via a sampling-based optimizer like the cross-entropy method) against the ensemble’s disagreement as a signal of uncertainty:
def plan_action_sequence(dynamics_ensemble, state, horizon, n_candidates=500):
candidates = sample_action_sequences(n_candidates, horizon)
scores = []
for actions in candidates:
# Roll out each candidate sequence through every ensemble member
trajectory_rewards = [rollout(model, state, actions) for model in dynamics_ensemble]
scores.append(np.mean(trajectory_rewards)) # penalize high disagreement implicitly
return candidates[np.argmax(scores)]
Averaging across an ensemble naturally discounts action sequences the models disagree about, which keeps the planner from confidently walking into regions the model has no real basis to predict well.
World Models: Learning and Planning Entirely in a Latent Space
World Models (Ha & Schmidhuber, 2018) compress raw observations with a VAE into a small latent vector, train an RNN to predict how that latent evolves over time, and train a tiny controller on top of the RNN’s hidden state — small enough that the controller itself can even be optimized with evolutionary strategies instead of gradients.
graph LR
A[Raw Observation] --> B[VAE Encoder]
B --> C[Latent State z]
C --> D[RNN Dynamics Model]
D --> E[Predicted Next Latent]
C --> F[Controller]
F --> G[Action]
G --> D
style D fill:#6366f1,color:#fff
The controller never has to process raw pixels directly — by the time training happens, the world has already been compressed into a small, predictable latent space.
MuZero: Planning With a Model That Doesn’t Know the Rules
AlphaZero needed a hand-coded simulator (the actual rules of Go or Chess) to run Monte Carlo Tree Search. MuZero (Schrittwieser et al., 2020) removes that requirement entirely: it learns a latent dynamics model end-to-end, trained only to be useful for predicting the value, policy, and reward — not to reconstruct real observations at all — and plans with MCTS directly over that learned latent model.
def muzero_mcts_step(latent_state, model, n_simulations=50):
root = Node(latent_state)
for _ in range(n_simulations):
node = root
path = []
while node.expanded():
action, node = select_child(node) # UCB-style selection
path.append(action)
value, reward, policy, next_latent = model.predict(node.latent_state, path[-1])
node.expand(policy, next_latent)
backpropagate(path, value)
return select_action_from_visit_counts(root)
The insight — plan over a model trained purely to be useful for planning, rather than trained to reconstruct the environment accurately — is what let MuZero match AlphaZero’s board-game performance while also generalizing to Atari, where no one hand-wrote the rules.
Dreamer: Learning Purely by “Imagination”
Dreamer (Hafner et al., 2020) pushes furthest: after learning a latent world model, the actor-critic policy is trained entirely on imagined rollouts generated by the model, with real environment interaction used only to keep the world model itself accurate and to collect fresh data — the policy itself never needs a real-environment gradient step.
Comparison Table
| Technique | What’s Learned | Planning Method | Key Idea |
|---|---|---|---|
| Dyna-Q | Tabular model | Extra Q-updates from simulated transitions | Every real step buys many free updates |
| PETS | Probabilistic model ensemble | Sampling-based planning (CEM) | Uncertainty-aware planning avoids model exploitation |
| World Models | VAE + RNN latent dynamics | Small controller on latent state | Compress the world before planning in it |
| MuZero | Latent dynamics (rules unknown) | MCTS over learned latent model | Model trained to be useful for planning, not accurate reconstruction |
| Dreamer | Latent dynamics + actor-critic | Policy trained on imagined rollouts | Learn the policy entirely inside imagination |
Applications
- Board games and perfect-information games — MuZero-style planning is the state of the art for Go, Chess, Shogi, and Atari without hand-coded rules.
- Robotics — PETS-style uncertainty-aware planning and Dreamer-style imagination training dramatically cut the number of expensive real-world rollouts needed.
- Any domain with an expensive or risky real environment — autonomous driving simulation, industrial control, and healthcare treatment policy research all favor model-based approaches for the same sample-efficiency reason.
Key Learnings
- The core trade is sample efficiency for model bias. Every technique here buys fewer real-environment interactions at the cost of being only as good as the learned model — which is why uncertainty-awareness (PETS) and learning the model to be plan-useful rather than accurate (MuZero) both matter so much.
- You don’t need to model the world accurately, just usefully. MuZero’s central lesson is that a latent model trained purely for downstream planning performance can outperform one trained to reconstruct reality.
- Model-based and model-free aren’t opposites — they compose. Dyna-Q is literally model-free Q-learning with extra simulated updates bolted on, and Dreamer’s imagined rollouts still train an ordinary actor-critic loss underneath.