Actor-Critic Methods Explained: From Vanilla Actor-Critic to A3C, DDPG, TD3, SAC, and PPO

TL;DR

Actor-critic methods pair a policy network (the actor) with a value network (the critic) so the actor gets a low-variance, step-by-step learning signal instead of waiting for a full episode to score itself. The lineage: vanilla Actor-Critic → batched A2C → asynchronous A3C → continuous-control DDPG → its stabilized successor TD3 → maximum-entropy SAC → trust-region TRPO/PPO, tied together by the GAE advantage estimator. Used in robotics control, autonomous driving policy research, large-scale game AI, financial portfolio allocation, and PPO specifically powers RLHF for LLM alignment.

Problem Statement

Pure policy-gradient methods (REINFORCE) learn a policy directly but suffer from high variance — you only find out if an action was good after an entire episode finishes and you sum up the total return. Pure value-based methods (Q-learning, covered in a companion article on this site) are low-variance but require taking a max over actions, which breaks down completely in continuous action spaces like robot joint torques or steering angles.

Actor-critic methods combine both: an actor (the policy π(a|s)) decides what to do, and a critic (a value function V(s) or Q(s,a)) evaluates how good that decision was, giving the actor a low-variance, step-by-step learning signal instead of waiting for the episode to end.

Vanilla Actor-Critic

The critic learns V(s) via ordinary TD learning; the actor updates its policy in the direction that increases the probability of actions that did better than the critic expected (the advantage, A = r + γV(s') - V(s)):

def actor_critic_step(actor, critic, s, a, r, s_next, log_prob, gamma=0.99):
    value = critic(s)
    next_value = critic(s_next).detach()
    td_target = r + gamma * next_value
    advantage = td_target - value

    critic_loss = advantage.pow(2)                  # regress V(s) toward TD target
    actor_loss = -log_prob * advantage.detach()      # push up log-prob of good actions

    return actor_loss, critic_loss

The advantage term is the whole point: instead of scaling the policy gradient by the raw (high-variance) return, you scale it by how much better than expected the outcome was — a much lower-variance signal.

A2C: Advantage Actor-Critic

A2C is the synchronous, batched version of this idea: multiple parallel environment copies collect a fixed number of steps each, advantages are computed for the whole batch, and one synchronized gradient update is applied. This is simply the vanilla algorithm made efficient enough to run on a GPU with many environments in lockstep.

A3C: Asynchronous Advantage Actor-Critic

A3C (Mnih et al., 2016) runs many actor-learner workers on separate CPU threads, each with its own copy of the environment and network, computing gradients locally and pushing them asynchronously to a shared global network.

graph TD
    G[Global Actor-Critic Network] --> W1[Worker 1: own env copy]
    G --> W2[Worker 2: own env copy]
    G --> W3[Worker 3: own env copy]
    G --> W4[Worker N: own env copy]
    W1 -->|async gradient push| G
    W2 -->|async gradient push| G
    W3 -->|async gradient push| G
    W4 -->|async gradient push| G

    style G fill:#6366f1,color:#fff

The asynchrony itself acts as a decorrelation mechanism (similar in spirit to DQN’s replay buffer) — each worker experiences a different trajectory at a different point in time, so the gradients averaged into the global network aren’t all correlated with each other. A2C later showed you get the same benefit synchronously without the engineering complexity of lock-free asynchronous updates.

DDPG: Deep Deterministic Policy Gradient

Everything above uses a stochastic policy over discrete or low-dimensional actions. DDPG (Lillicrap et al., 2016) targets continuous control by learning a deterministic actor μ(s) alongside a Q(s,a) critic, borrowing DQN’s replay buffer and target networks:

def ddpg_actor_loss(actor, critic, states):
    actions = actor(states)
    return -critic(states, actions).mean()   # move actor toward actions the critic rates highest

def ddpg_critic_loss(actor_target, critic, critic_target, batch, gamma=0.99):
    s, a, r, s_next, done = batch
    next_action = actor_target(s_next)
    target_q = r + gamma * critic_target(s_next, next_action) * (1 - done)
    return F.mse_loss(critic(s, a), target_q.detach())

Because the actor is deterministic, exploration has to be added externally — DDPG adds noise (originally Ornstein-Uhlenbeck, later simple Gaussian) directly to the actions taken during training.

TD3: Twin Delayed DDPG

DDPG inherited Q-learning’s overestimation bias problem and is notoriously unstable to tune. TD3 (Fujimoto et al., 2018) fixes it with three changes:

1. Twin critics — learn two independent Q networks and take the minimum of the two when computing the target, directly countering overestimation (the continuous-action analogue of Double DQN).
2. Delayed policy updates — update the actor (and target networks) less frequently than the critics, letting the value estimate settle before the policy chases it.
3. Target policy smoothing — add clipped noise to the target action, smoothing the Q-function so the policy can’t exploit sharp, erroneous peaks in the critic’s estimate.

SAC: Soft Actor-Critic

SAC (Haarnoja et al., 2018) reframes the objective entirely as maximum-entropy RL: maximize expected return plus the entropy of the policy, so the agent is rewarded for acting as randomly as possible subject to still performing well.

J(π) = E[ Σ γ^t ( r_t + α · H(π(·|s_t)) ) ]

This built-in entropy bonus gives SAC exploration that falls out of the objective itself (no external noise process needed like DDPG/TD3), and the temperature α controlling the exploration/exploitation tradeoff can itself be learned automatically rather than hand-tuned. SAC also uses twin critics like TD3, and is widely regarded as the most sample-efficient and stable of the continuous-control actor-critic methods.

TRPO and PPO: Trust-Region Policy Optimization

A separate failure mode actor-critic methods face: a single large policy update can catastrophically collapse performance, because the policy generating your data and the policy you’re updating drift apart. TRPO (Schulman et al., 2015) constrains each update to stay within a KL-divergence “trust region” of the old policy, solved via constrained optimization. PPO (Schulman et al., 2017) achieves nearly the same effect far more simply, by clipping the probability ratio between new and old policy directly in the loss:

def ppo_clipped_loss(new_log_probs, old_log_probs, advantages, epsilon=0.2):
    ratio = torch.exp(new_log_probs - old_log_probs)
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - epsilon, 1 + epsilon) * advantages
    return -torch.min(unclipped, clipped).mean()

By taking the minimum of the clipped and unclipped objective, PPO removes the incentive to push the policy ratio far outside [1-ε, 1+ε] in either direction, giving TRPO-like stability with plain first-order gradient descent — no second-order constrained optimization required. This simplicity is why PPO became the default actor-critic algorithm across both robotics and, notably, LLM fine-tuning.

GAE: The Advantage Estimator Tying It Together

Most modern actor-critic methods (including PPO) compute advantages with Generalized Advantage Estimation (Schulman et al., 2016), which blends multi-step TD errors with an exponential weighting factor λ, trading off bias against variance in a single tunable knob — λ=0 recovers the low-variance, high-bias one-step TD advantage; λ=1 recovers the high-variance, unbiased Monte Carlo advantage.

Summary Table

Applications

• Robotics control — DDPG, TD3, and SAC are the standard choices for continuous joint-torque and locomotion control, both in simulation and sim-to-real transfer.
• Autonomous driving policy research — continuous steering/throttle control fits the same continuous-action actor-critic framing.
• Game AI at scale — A3C-style asynchronous training and PPO have both been used in large-scale game-playing agents (real-time strategy, first-person environments).
• RLHF for LLMs — PPO is the algorithm behind reinforcement learning from human feedback used to align large language models, treating the language model itself as the actor and a learned reward model’s score as the critic’s target signal.
• Financial portfolio allocation — continuous position-sizing decisions are a natural fit for SAC/TD3-style continuous control.

Key Learnings

1. The actor-critic family exists to cut variance without losing the ability to handle continuous actions — the critic is what makes this possible; every method above is a different way of making that critic (and the actor’s use of it) more stable or more sample-efficient.
2. Continuous control and stability are the two axes of progress. DDPG solved continuous actions but was unstable; TD3 and SAC solved stability; PPO solved the separate problem of large destructive policy updates without going to costly second-order optimization.
3. PPO’s dominance is about simplicity, not just performance. It gets most of TRPO’s stability guarantees with a one-line change to the loss function, which is why it became the practical default far beyond robotics — including as the core RL algorithm behind modern LLM alignment pipelines.