← Back to all posts
Reinforcement LearningPOMDPBelief StateDRQN

Reinforcement Learning Under Partial Observability: POMDPs, Belief States, and DRQN

How RL handles environments where the true state isn't directly visible — the POMDP formalism, exact belief-state tracking, and Deep Recurrent Q-Networks that learn to summarize observation history implicitly — with code and applications in robotics and imperfect-information games.

O Ohara University · · 11 min read

TL;DR

Every method covered elsewhere on this site assumes the agent can directly observe the true state — the Markov assumption. Real sensors are noisy, occluded, or simply don’t reveal everything that matters (a self-driving car can’t see around a blind corner; a poker player can’t see opponents’ cards). A POMDP (Partially Observable MDP) formalizes this gap between the true state and what the agent actually observes. The classical fix is tracking an explicit belief state — a probability distribution over what the true state might be — updated via Bayesian filtering; the modern deep RL fix is DRQN, which replaces explicit belief tracking with a recurrent network that learns to summarize observation history implicitly, without ever computing a belief distribution by hand. Used in robotics with imperfect sensing, imperfect-information games, and any domain with occlusion or sensor noise.

The Belief-State Update Loop

graph LR
    A[Prior Belief] --> B[Take Action]
    B --> C[Receive Observation]
    C --> D[Bayesian Belief Update]
    D -->|new belief becomes the prior| A

    style D fill:#6366f1,color:#fff

Problem Statement

Q-learning, actor-critic, and every other technique on this site rely on the Markov property: the current state alone is enough to make an optimal decision, with no benefit from remembering history. When the agent only receives a partial observation o instead of the true state s, that property breaks — the optimal action can genuinely depend on everything observed so far, not just the current observation, because past observations carry information about the hidden true state that the current observation alone doesn’t reveal.

The POMDP Formalism

A POMDP extends an MDP with an observation function p(o|s) describing what the agent perceives given the true (hidden) state, on top of the usual transition and reward functions. The agent never sees s directly — only a stream of observations o_1, o_2, ... it must use to infer enough about s to act well.

class POMDP:
    def step(self, true_state, action):
        next_true_state = self.transition_fn(true_state, action)     # true dynamics, hidden from agent
        reward = self.reward_fn(true_state, action)
        observation = self.observation_fn(next_true_state)            # what the agent actually perceives
        return next_true_state, observation, reward

Belief States: Exact Bayesian Tracking

The classical solution maintains a belief state b(s) — a full probability distribution over which true state the agent is likely in, updated after every action and observation via Bayes’ rule:

def belief_update(belief, action, observation, transition_model, observation_model):
    # Predict: propagate the belief forward through the (uncertain) transition dynamics
    predicted_belief = {
        s_next: sum(belief[s] * transition_model(s, action, s_next) for s in belief)
        for s_next in belief
    }

    # Correct: reweight by how likely this observation is under each possible resulting state
    unnormalized = {s: predicted_belief[s] * observation_model(s, observation) for s in predicted_belief}
    normalization = sum(unnormalized.values())
    return {s: p / normalization for s, p in unnormalized.items()}

Crucially, a POMDP’s belief state is itself Markovian — the belief distribution alone, without needing the entire raw observation history, is a sufficient statistic for optimal decision-making. This turns a POMDP into an ordinary (but much larger, continuous-valued) MDP over belief states, solvable in principle with the dynamic programming methods covered in the companion article — in practice, exact belief tracking is only tractable for small, discrete state spaces, since the belief itself is a full probability distribution over every possible true state.

DRQN: Learning to Summarize History Implicitly

Computing an exact belief update requires knowing the transition and observation models explicitly — unavailable in most deep RL settings, and intractable at the scale of raw pixel observations regardless. DRQN (Deep Recurrent Q-Network, Hausknecht & Stone, 2015) sidesteps explicit belief tracking entirely: replace DQN’s feedforward network with a recurrent one (LSTM/GRU), and let the network’s hidden state learn, purely from the training signal, whatever implicit summary of observation history turns out to be useful for the task.

class DRQN(nn.Module):
    def __init__(self, obs_dim, hidden_dim, n_actions):
        super().__init__()
        self.encoder = nn.Linear(obs_dim, hidden_dim)
        self.lstm = nn.LSTMCell(hidden_dim, hidden_dim)
        self.q_head = nn.Linear(hidden_dim, n_actions)

    def forward(self, observation, hidden_state):
        encoded = F.relu(self.encoder(observation))
        hidden_state = self.lstm(encoded, hidden_state)   # hidden state carries history forward
        q_values = self.q_head(hidden_state[0])
        return q_values, hidden_state

No one designs what the hidden state should represent — over training, it comes to implicitly encode whatever aspects of history turn out to matter for predicting good actions, which can end up resembling a learned, task-specific approximation of a belief state without ever computing one explicitly. This is the same recurrent-memory idea used in R2D2 (covered in the companion distributed RL article) and in RL² (covered in the companion meta-RL article) — a recurring pattern across the field: when explicit state or belief tracking is intractable, let a recurrent network’s hidden state learn to summarize what’s needed instead.

Comparison Table

ApproachState RepresentationRequires Known Models?Scales to High-Dimensional Observations?
Exact Belief TrackingFull probability distribution over true statesYes (transition + observation models)No, intractable at scale
DRQNImplicit, learned recurrent hidden stateNoYes

Applications

Key Learnings

  1. Partial observability isn’t a corner case — most real-world RL problems have it to some degree. Any sensor is imperfect, which means the clean Markov assumption underlying Q-learning and actor-critic methods is always an approximation in practice, just often a good enough one.
  2. The belief state is the theoretically clean answer, but it doesn’t scale. Exact Bayesian belief tracking turns a POMDP into a solvable (if large) MDP, but computing it requires known models and tractable state spaces — exactly the conditions deep RL usually doesn’t have.
  3. Recurrence is deep RL’s practical substitute for explicit belief tracking, and this pattern — a hidden state implicitly learning to summarize whatever history matters — reappears directly in R2D2’s distributed recurrent replay and RL²’s meta-learning, both covered in companion articles on this site.

References