Multi-Agent Reinforcement Learning: From Independent Learners to QMIX and Self-Play

TL;DR

Multi-agent RL adds a problem single-agent RL never faces: every other agent is also learning and changing at the same time, so the environment is non-stationary from any one agent’s point of view. Independent Q-Learning ignores the problem and often fails to converge; MADDPG fixes it with a centralized critic that sees everyone’s actions during training while each actor stays decentralized at execution time; VDN and QMIX solve the credit-assignment problem in cooperative teams by decomposing a joint value function into per-agent pieces; and self-play sidesteps non-stationarity entirely by having an agent train against snapshots of itself. Used in game AI (Dota 2, StarCraft), traffic signal coordination, robotic swarms, and multi-agent autonomous vehicle coordination.

Problem Statement

Every algorithm in the companion Q-learning and actor-critic articles on this site assumes a stationary environment: the same state leads to roughly the same outcome distribution across training. With multiple learning agents, that assumption breaks — from any single agent’s perspective, the “environment” includes the other agents’ policies, which are shifting under their own training the entire time. A second problem shows up specifically in cooperative teams: if the team gets one shared reward, how do you credit which agent’s action actually mattered?

Independent Q-Learning: The Naive Baseline

The simplest approach: give each agent its own independent Q-learning agent, oblivious to the others, treating everyone else as part of the environment.

def independent_q_learning_update(agents, joint_state, joint_action, joint_reward, joint_next_state):
    for i, agent in enumerate(agents):
        agent.q_learning_update(
            s=joint_state[i], a=joint_action[i],
            r=joint_reward[i], s_next=joint_next_state[i],
        )

This can work in simple settings, but as other agents’ policies shift, an agent’s own learned Q-values become stale almost immediately — the non-stationarity problem in its purest form. It’s the standard baseline every other technique in this article is trying to beat.

MADDPG: Centralized Training, Decentralized Execution

MADDPG (Lowe et al., 2017) resolves non-stationarity with a trick that’s now standard across multi-agent RL: give each agent’s critic access to every agent’s observations and actions during training (so the critic sees a stationary joint environment), while each agent’s actor only ever sees its own local observation, so execution stays fully decentralized.

def maddpg_critic_loss(critic_i, all_actors_target, all_critics_target, batch, gamma=0.99):
    obs, actions, rewards, next_obs, dones = batch
    next_actions = [actor(next_obs[j]) for j, actor in enumerate(all_actors_target)]

    target_q = rewards[i] + gamma * critic_i.target(
        torch.cat(next_obs, dim=-1), torch.cat(next_actions, dim=-1)
    ) * (1 - dones)

    current_q = critic_i(torch.cat(obs, dim=-1), torch.cat(actions, dim=-1))
    return F.mse_loss(current_q, target_q.detach())

Because the critic conditions on everyone’s actions, it can correctly attribute how good agent i’s action was given what everyone else did — from the critic’s point of view, the joint system is stationary even though no individual agent’s policy is fixed.

VDN and QMIX: Value Decomposition for Cooperative Teams

In cooperative settings with one shared team reward, MADDPG’s per-agent critics still don’t answer “which agent’s action actually deserves credit?” VDN (Value Decomposition Networks, Sunehag et al., 2017) takes the simplest possible approach: assume the joint Q-value is just the sum of independent per-agent Q-values, and train the sum against the shared reward.

Q_total(s, a₁, ..., aₙ) = Σᵢ Qᵢ(sᵢ, aᵢ)

QMIX (Rashid et al., 2018) generalizes this with a learned, more expressive mixing network instead of a plain sum — constrained so that the mixing weights stay non-negative, guaranteeing that whatever action maximizes each individual Qᵢ also maximizes the joint Q_total (a property called monotonicity), which is what makes decentralized greedy execution still correct for the team.

class QMixer(nn.Module):
    def forward(self, agent_qs, global_state):
        # Hypernetwork produces non-negative mixing weights conditioned on global state
        w1 = torch.abs(self.hyper_w1(global_state)).view(-1, self.n_agents, self.mixing_dim)
        b1 = self.hyper_b1(global_state)
        hidden = F.elu(torch.bmm(agent_qs.unsqueeze(1), w1) + b1)

        w2 = torch.abs(self.hyper_w2(global_state)).view(-1, self.mixing_dim, 1)
        b2 = self.hyper_b2(global_state)
        return torch.bmm(hidden, w2) + b2   # Q_total

The non-negativity constraint on mixing weights is the whole trick — it’s what lets each agent still act greedily on its own local Qᵢ at execution time while guaranteeing that’s consistent with maximizing the team’s joint value.

Self-Play: Sidestepping Non-Stationarity Entirely

Rather than solving non-stationarity, self-play exploits it: an agent trains against copies (often historical snapshots) of itself, so the “other agent” in the environment is always at a similar skill level, creating an automatic curriculum that escalates in difficulty as the agent improves. This is the core technique behind AlphaZero’s self-play against its own past versions, and behind large-scale competitive game agents like OpenAI Five and AlphaStar, which maintained a league of past policy snapshots to train against for diversity and to avoid overfitting to any one opponent style.

Comparison Table

Applications

• Competitive game AI — Dota 2 (OpenAI Five), StarCraft II (AlphaStar), and poker bots all rely heavily on self-play and league training.
• Cooperative robotics swarms — coordinating warehouse robots or drone fleets toward a shared objective fits the QMIX/VDN cooperative framing directly.
• Traffic signal control — each intersection as an agent, cooperating (implicitly or via shared reward) to minimize city-wide congestion.
• Autonomous vehicle coordination — merging, intersection negotiation, and platooning are naturally multi-agent problems with both cooperative and competitive elements.

Key Learnings

1. Non-stationarity is the defining problem of multi-agent RL, and almost every technique here is a different strategy for coping with it — centralize training (MADDPG), decompose value functions (VDN/QMIX), or make the opponent’s skill track your own (self-play).
2. Centralized training with decentralized execution is now close to a default design pattern, not just a MADDPG-specific trick — it reappears across cooperative and competitive multi-agent methods because it’s the cleanest way to get a stationary training signal without sacrificing deployability.
3. Value decomposition only works because of the monotonicity constraint. Without QMIX’s non-negative mixing weights, there’d be no guarantee that agents acting greedily and independently at execution time actually maximizes the team’s joint reward.