Goal-Conditioned RL and Hindsight Experience Replay: Learning From Every Failure

TL;DR

Most RL on this site trains one policy for one fixed objective. Goal-conditioned RL instead trains a single policy π(a|s,g) that generalizes across an entire space of goals g, using Universal Value Function Approximators (UVFA) to make the goal a first-class input alongside the state. This matters enormously for sparse-reward tasks — “did the robotic arm reach this exact target position” almost never fires early in training, so there’s barely any learning signal. Hindsight Experience Replay (HER) solves that with a strikingly simple trick: after a failed episode, relabel it in the replay buffer as if the goal had been whatever state was actually reached — turning every failure into a successful example for some goal, even if not the one originally intended. Used in robotic manipulation, navigation, and any sparse-reward task where “did I reach the target” is the only natural reward signal.

Turning a Failure Into a Success

graph LR
    A[Episode Fails to Reach Intended Goal] --> B[Relabel Goal as State Actually Reached]
    B --> C[Now a Successful Example]
    C --> D[Store in Replay Buffer Alongside Original]

    style C fill:#6366f1,color:#fff

Problem Statement

A robotic arm learning to reach an arbitrary target position with a binary reward (“1 if within 5cm of the target, else 0”) almost never receives a positive reward during early random exploration — the arm essentially never happens to end up exactly where a randomly sampled target says it should. Reward shaping (giving partial credit for getting closer) can help, but it requires hand-designing a distance metric that isn’t always meaningful (closer in joint-angle space isn’t always closer in the way that matters), and it can bias the policy toward the shaped proxy rather than the real objective. Goal-conditioned RL and HER attack the sparse-reward problem from a different angle entirely: making the goal itself something the training process can exploit.

Universal Value Function Approximators: Goal as an Input

The straightforward extension: instead of learning Q(s,a), learn Q(s,a,g), with the goal concatenated into the network’s input alongside the state.

class GoalConditionedQNetwork(nn.Module):
    def __init__(self, state_dim, goal_dim, action_dim, hidden_dim=256):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(state_dim + goal_dim, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, action_dim),
        )

    def forward(self, state, goal):
        return self.network(torch.cat([state, goal], dim=-1))

This alone doesn’t solve the sparse-reward problem — it just makes it possible for one network to represent the value of reaching many different goals, generalizing across goals the same way a normal network generalizes across states. It’s HER’s relabeling trick, layered on top, that actually produces usable learning signal from otherwise-failed episodes.

Hindsight Experience Replay: Relabeling Failure as Success

HER (Andrychowicz et al., 2017) observes that even a “failed” episode — the arm ended up somewhere other than the intended target — is a perfectly good demonstration of reaching whatever state it actually ended up at. So, when storing a trajectory in the replay buffer, HER stores it multiple times: once with the original goal (usually still a failure), and again with the goal relabeled to a state the trajectory actually achieved (now a guaranteed success).

def her_relabel_trajectory(trajectory, reward_fn, strategy="future", k=4):
    relabeled_transitions = []

    for t, (state, action, next_state, original_goal, done) in enumerate(trajectory):
        # Always keep the original-goal transition
        relabeled_transitions.append((state, action, next_state, original_goal,
                                       reward_fn(next_state, original_goal), done))

        # Additionally sample k "hindsight" goals from states the trajectory actually reached
        future_states = trajectory[t:]  # "future" strategy: sample from later in this same episode
        for _ in range(min(k, len(future_states))):
            hindsight_goal = random.choice(future_states).next_state
            hindsight_reward = reward_fn(next_state, hindsight_goal)  # near-guaranteed success for itself
            relabeled_transitions.append((state, action, next_state, hindsight_goal, hindsight_reward, done))

    return relabeled_transitions

The “future” strategy — sampling hindsight goals from later states in the same episode — is the standard choice, since it guarantees the relabeled goal was genuinely reachable from the current state under the actual policy that generated the trajectory, giving the network dense, realistic learning signal even when the original, intended goal was never achieved even once.

Why This Actually Works

The insight isn’t just “get more data” — it’s that a goal-conditioned Q-function trained on relabeled successes learns the general skill of goal-reaching, which transfers to the originally intended goals too, because the network can’t tell from its own weights which goals were “real” targets and which were hindsight relabels — it just learns Q(s,a,g) as a genuinely general function of any g, real or relabeled:

def her_training_step(q_network, replay_buffer, batch_size=256, gamma=0.99):
    batch = replay_buffer.sample(batch_size)   # buffer contains both original and relabeled transitions
    states, actions, next_states, goals, rewards, dones = batch

    with torch.no_grad():
        next_q = q_network(next_states, goals).max(dim=-1).values
        td_target = rewards + gamma * next_q * (1 - dones)

    current_q = q_network(states, goals).gather(-1, actions)
    return F.mse_loss(current_q, td_target)

Any off-policy algorithm covered elsewhere on this site — DQN, DDPG, SAC — can be combined with HER essentially as a drop-in replay buffer modification; HER is an augmentation to the data, not a new policy-optimization algorithm in its own right.

Comparison Table

Applications

• Robotic manipulation with sparse rewards — reaching, pushing, and grasping tasks specified by a target position or configuration are the canonical HER use case, since “did I reach the exact target” is naturally sparse.
• Navigation to arbitrary targets — a single goal-conditioned policy that navigates to any specified location, rather than training a separate policy per destination.
• Multi-task robotics — one goal-conditioned policy can substitute for what would otherwise require training and maintaining many separate single-task policies.
• Any sparse binary-reward task with a well-defined “achieved state” — the relabeling trick generalizes to any domain where you can meaningfully ask “what goal would this trajectory have succeeded at?”

Key Learnings

1. HER’s core move — redefining failure as success for a different goal — is a genuinely different strategy than reward shaping, not just an alternative to it. Reward shaping tries to make the original sparse signal denser; HER instead exploits the goal-conditioned structure of the problem to manufacture new, genuinely dense signal without changing the task’s true objective at all.
2. Goal-conditioning and hindsight relabeling are two separate ideas that only pay off combined. A goal-conditioned network without HER has no more learning signal than an unconditioned one; HER’s relabeling without goal-conditioning has nothing to relabel into — a fixed-goal policy can’t benefit from data about a different goal.
3. This pattern generalizes beyond literal spatial goals. Any task where “success” can be redefined post-hoc based on what actually happened — not just reaching a physical target — is a candidate for the same hindsight-relabeling trick, which is part of why it’s become a standard tool well beyond its original robotic manipulation setting.

References

• Schaul, T. et al. (2015). Universal Value Function Approximators. ICML.
• Andrychowicz, M. et al. (2017). Hindsight Experience Replay.