Hierarchical Reinforcement Learning: From the Options Framework to Feudal Networks

TL;DR

Flat RL treats every problem as a sequence of primitive, single-timestep actions, which struggles badly on long-horizon tasks with sparse rewards — credit assignment across thousands of steps is nearly hopeless. Hierarchical RL fixes this with temporal abstraction: the Options framework formalizes reusable, multi-step sub-policies; the Option-Critic architecture learns those sub-policies end-to-end with gradients instead of hand-designing them; Feudal Networks split the agent into a slow manager that sets goals and a fast worker that pursues them; and HIRO makes that manager-worker split sample-efficient with off-policy correction. Used in long-horizon robotics manipulation, sparse-reward games like Montezuma’s Revenge, and hierarchical navigation tasks.

Problem Statement

Reward is often sparse and delayed — a robot assembling a product only gets rewarded on final success, dozens or hundreds of primitive motor actions later. Flat RL has to propagate that single reward signal backward through every one of those steps, which is slow and often fails outright in practice (Montezuma’s Revenge became the canonical example of a game flat DQN essentially couldn’t solve). Hierarchical RL restructures the problem: instead of one long sequence of primitive actions, learn temporally-extended “skills” and a higher-level policy that sequences them.

The Options Framework

The Options framework (Sutton, Precup, Singh, 1999) formalizes a temporally-extended action, or option, as three components: an initiation set (states where the option can start), an internal policy (what to do while the option runs), and a termination condition (when to hand control back). A high-level policy then selects among options the same way a flat policy selects among primitive actions, but each choice can span many environment steps.

class Option:
    def __init__(self, initiation_set, policy, termination_fn):
        self.initiation_set = initiation_set
        self.policy = policy
        self.termination_fn = termination_fn

    def execute(self, env, state):
        trajectory = []
        while state in self.initiation_set and not self.termination_fn(state):
            action = self.policy(state)
            state, reward, done = env.step(action)
            trajectory.append((state, action, reward))
            if done:
                break
        return trajectory

This turns the decision problem into a semi-Markov decision process over options rather than primitive actions — the high-level policy’s effective time horizon shrinks dramatically, since each decision now covers many environment steps at once.

Option-Critic: Learning Options End-to-End

The original Options framework typically required options to be hand-designed. Option-Critic (Bacon, Harb, Precup, 2017) instead learns the option policies, termination conditions, and the high-level option-selection policy all jointly with gradient descent, using a policy-gradient-style theorem extended to options:

def option_critic_gradients(option_policy, termination_fn, critic, state, option, action):
    advantage = critic.q_option_action(state, option, action) - critic.q_option(state, option)
    policy_grad = -option_policy.log_prob(action) * advantage.detach()

    termination_advantage = critic.q_option(state, option) - critic.value(state)
    termination_grad = termination_fn(state) * termination_advantage.detach()

    return policy_grad, termination_grad

No one needs to hand-specify what the sub-skills should be — the network discovers useful, reusable temporal abstractions purely from the training signal, the same way convolutional filters emerge from image classification training without being hand-designed.

Feudal Networks: A Manager-Worker Hierarchy

FeUdal Networks (Vezhnevets et al., 2017) take a different structural approach, inspired by feudal management hierarchies: a slow-ticking Manager operates in a compressed latent space and outputs a goal direction every few timesteps, and a fast-ticking Worker receives that goal and picks primitive actions every single timestep to move the internal state in the direction the Manager specified.

graph TD
    A[Environment State] --> B[Shared Perception Module]
    B --> C[Manager - operates every c steps]
    B --> D[Worker - operates every step]
    C -->|goal vector| D
    D --> E[Primitive Action]
    E --> A

    style C fill:#6366f1,color:#fff
    style D fill:#10b981,color:#fff

Crucially, the Manager is never trained to predict primitive actions or rewards directly — it’s trained to set goals that, in hindsight, corresponded to directions of real state-space progress, which is what forces it to learn abstractions at a genuinely different timescale than the Worker.

HIRO: Making the Hierarchy Sample-Efficient

Feudal-style architectures are naturally off-policy-unfriendly: by the time the Worker has acted on a goal, the Manager’s policy for choosing that goal may have already changed, making old (state, goal, outcome) tuples stale for replay. HIRO (Nachum et al., 2018) fixes this with an off-policy correction — it relabels past goals in the replay buffer with whichever goal would have been most likely to produce the actual observed worker behavior, letting the manager reuse old experience efficiently the same way Hindsight Experience Replay reuses failed trajectories.

Comparison Table

Applications

• Sparse-reward games — Montezuma’s Revenge and similar long-horizon exploration benchmarks are the standard testbed hierarchical RL was built to solve.
• Long-horizon robotic manipulation — assembly and multi-step manipulation tasks where the final reward is far removed from any single motor command.
• Hierarchical navigation — high-level waypoint planning (Manager) combined with low-level obstacle avoidance and locomotion control (Worker).

Key Learnings

1. Temporal abstraction is fundamentally a credit-assignment fix. Every technique here exists to shrink the effective decision horizon a high-level policy has to reason over, which is what makes sparse, delayed rewards tractable.
2. Hand-designed sub-skills don’t scale, which is why the field moved toward end-to-end learned hierarchies. Option-Critic and Feudal Networks both discover reusable temporal structure directly from the training signal rather than requiring a human to specify what the “skills” should be.
3. Off-policy sample efficiency in hierarchical RL isn’t automatic — it has to be engineered. HIRO’s goal-relabeling trick is a reminder that stacking a second learning problem (the manager) on top of a first (the worker) introduces its own non-stationarity that needs a dedicated fix.