Federated Reinforcement Learning: Training Across Devices That Can't Share Data

TL;DR

Every distributed RL architecture covered in a companion article (IMPALA, Ape-X) assumes all actors’ data can be pooled centrally. Federated RL removes that assumption: many agents or devices — each interacting with their own local environment — train collaboratively without ever sharing raw trajectories, states, or observations, only sharing model updates (weights or gradients) that get aggregated centrally. This matters whenever raw experience is privacy-sensitive (personal devices, hospitals, individual robots with proprietary data) or simply too large to transmit. Federated averaging, adapted from supervised federated learning, is the standard aggregation mechanism, but RL introduces a problem supervised federated learning doesn’t face as sharply: each agent’s local environment can be genuinely different (non-iid), which can make naive averaging actively harmful rather than just noisy.

A Federated Training Round

graph TD
    A[Global Policy] --> B[Agent 1: local env, local training]
    A --> C[Agent 2: local env, local training]
    A --> D[Agent N: local env, local training]
    B -->|weights only, no raw data| E[Central Aggregation]
    C -->|weights only, no raw data| E
    D -->|weights only, no raw data| E
    E --> A

    style E fill:#6366f1,color:#fff

Problem Statement

A fleet of household robots, a network of hospitals training a treatment policy, or a set of mobile devices personalizing a recommendation policy all share a common constraint: the raw experience each one generates (a robot’s camera feed, a hospital’s patient records, a user’s on-device behavior) can’t leave its local environment for privacy, bandwidth, or regulatory reasons. But training a single, capable shared policy from scratch on any one device’s limited local data alone would be far less sample-efficient than pooling everyone’s experience — federated RL is the attempt to get collaborative training’s sample-efficiency benefit without ever centralizing the raw data itself.

Federated Averaging for RL

The core mechanism, adapted directly from supervised federated learning (McMahan et al., 2017): each agent trains locally on its own environment for several steps, then only the resulting model parameters — not any trajectories or raw data — are sent to a central server, which averages them (typically weighted by how much local data each agent used) into a new global model that’s redistributed back out.

def federated_rl_round(global_policy, local_agents, local_steps=1000):
    local_updates = []
    for agent in local_agents:
        local_policy = copy.deepcopy(global_policy)
        train_locally(local_policy, agent.local_env, n_steps=local_steps)  # ordinary RL, e.g. PPO
        local_updates.append((local_policy.state_dict(), agent.n_local_samples))

    total_samples = sum(n for _, n in local_updates)
    averaged_state_dict = {}
    for key in global_policy.state_dict():
        averaged_state_dict[key] = sum(
            (n / total_samples) * state_dict[key] for state_dict, n in local_updates
        )

    global_policy.load_state_dict(averaged_state_dict)
    return global_policy

No agent ever transmits a single state, action, or reward from its local environment — only the numerical weights of its locally-trained network, which is what gives federated RL its core privacy property.

The Non-IID Problem: RL’s Extra Challenge

Supervised federated learning already has to handle non-identically-distributed data across clients (different users have different writing styles, different photo collections). RL’s version of this problem is often more severe: each agent’s environment itself can differ — different room layouts for household robots, different patient populations for hospitals, different network conditions for connected devices — meaning locally optimal policies can point in genuinely conflicting directions, not just noisy variations of the same underlying policy.

def federated_rl_with_personalization(global_policy, local_agents, local_steps, personalization_weight=0.3):
    updated_global = federated_rl_round(global_policy, local_agents, local_steps)

    personalized_policies = {}
    for agent in local_agents:
        # Blend the global (collaboratively trained) policy with a locally fine-tuned one
        personalized = copy.deepcopy(updated_global)
        fine_tune_locally(personalized, agent.local_env, n_steps=local_steps // 4)
        personalized_policies[agent.id] = interpolate_policies(
            updated_global, personalized, weight=personalization_weight
        )
    return updated_global, personalized_policies

Rather than forcing every agent to converge to one identical global policy, this keeps the global model as a strong shared starting point (benefiting from everyone’s collective experience) while still allowing each agent’s deployed policy to specialize to its own local environment’s quirks — a direct echo of the personalization use case in the companion meta-RL article, but achieved through federated collaboration instead of a meta-learned fast-adaptation initialization.

Comparison Table

Applications

• Fleets of robots or IoT devices — each device trains on its own local environment and contributes to a shared global policy without transmitting raw sensor data.
• Healthcare treatment policy research — multiple hospitals collaboratively training a treatment policy without any patient data leaving its home institution, a hard regulatory requirement in most jurisdictions.
• On-device personalization — mobile keyboards, recommendation systems, and similar applications where user behavior data is sensitive enough that it shouldn’t leave the device, but a globally shared model still benefits from everyone’s collective experience.
• Multi-site industrial control — factories or plants with similar but not identical processes, where sharing raw operational data across sites may be commercially sensitive even within the same company.

Key Learnings

1. Federated RL’s privacy property comes specifically from never transmitting raw experience, only model parameters — this is a strong and useful guarantee, but it’s not automatically total privacy (model weights can still leak information about training data in principle, which is a separate, ongoing research concern in federated learning generally).
2. Non-iid environments are a harder problem in RL than in supervised federated learning, because an RL agent’s entire environment (not just its local data distribution) can genuinely differ from another agent’s — plain averaging can push a policy toward a compromise that’s actually bad for everyone, rather than just being noisy.
3. Personalization on top of federation, not federation alone, is often the practical answer. Treating the global federated model as a strong shared starting point that each agent then locally fine-tunes — rather than expecting one single global policy to be optimal everywhere — resolves much of the non-iid tension in practice.

References

• McMahan, B. et al. (2017). Communication-Efficient Learning of Deep Networks from Decentralized Data.
• Zhuo, H. et al. (2019). Federated Reinforcement Learning.