Offline Reinforcement Learning: From Batch-Constrained Q-Learning to Decision Transformer

TL;DR

Offline RL learns a policy from a fixed, previously-collected dataset with zero further environment interaction — no exploration, no trial and error. Naive Q-learning fails badly here because it extrapolates confidently into actions the dataset never covers. BCQ fixes this by constraining the policy to actions similar to what’s in the data; CQL penalizes the Q-function directly for overestimating unseen actions; IQL avoids the problem altogether by never querying out-of-distribution actions in the first place; and Decision Transformer reframes the entire problem as sequence modeling, dropping the Bellman equation entirely. Used in healthcare treatment policy research, recommendation systems trained on logged interaction data, and robotics/autonomous driving where online exploration is too costly or dangerous.

Problem Statement

Every method in the companion Q-learning and actor-critic articles assumes you can keep collecting new experience by interacting with the environment. That assumption fails in high-stakes domains — you can’t let an untrained policy “explore” by prescribing random medical treatments or driving a car randomly to see what happens. Offline RL learns entirely from a static, previously-logged dataset instead, which sounds like ordinary off-policy learning but turns out to fail in a specific and severe way.

Why Naive Q-Learning Fails Offline

Standard Q-learning’s max over next-state actions doesn’t care whether the dataset actually contains examples of that action being taken. With no further environment interaction to correct it, the Q-function’s overestimation on unseen, out-of-distribution actions compounds with every bootstrapped update instead of being corrected the way it would be with fresh online data — this is extrapolation error, and it’s the central problem the entire offline RL literature exists to solve.

BCQ: Batch-Constrained Q-Learning

BCQ (Fujimoto et al., 2019) constrains the policy to only select actions that a generative model, trained on the dataset, judges as plausible given the data — instead of letting the policy consider the full action space, it perturbs and re-ranks a small set of dataset-similar candidate actions.

def bcq_action_selection(state, generator_model, perturbation_model, q_network, n_candidates=10):
    candidate_actions = generator_model.sample(state, n=n_candidates)   # actions like those in the data
    perturbed = [perturbation_model(state, a) for a in candidate_actions]  # small learned corrections

    q_values = [q_network(state, a) for a in perturbed]
    return perturbed[np.argmax(q_values)]

By construction, the policy can never select an action far outside what the dataset demonstrated, which directly caps how badly the Q-function’s extrapolation error can hurt it.

CQL: Conservative Q-Learning

CQL (Kumar et al., 2020) takes a different approach: instead of constraining which actions the policy can pick, it adds a regularization term to the Q-learning loss that explicitly pushes down the Q-values of actions not seen in the dataset, while pushing up the Q-values of actions actually taken:

def cql_loss(q_network, states, actions, td_target, alpha=1.0):
    standard_td_loss = F.mse_loss(q_network(states, actions), td_target)

    # Penalize the log-sum-exp over all actions (overestimated OOD actions dominate this term)
    all_action_q = q_network.q_over_all_actions(states)
    conservative_penalty = torch.logsumexp(all_action_q, dim=-1).mean() - q_network(states, actions).mean()

    return standard_td_loss + alpha * conservative_penalty

This gives a Q-function that’s a provable lower bound on the true value for out-of-distribution actions, so even if the policy is later tempted to exploit an overestimated action, there’s nothing overestimated left to exploit.

IQL: Implicit Q-Learning

Both BCQ and CQL still need to evaluate or query the Q-function at actions outside the dataset during training. IQL (Kostrikov et al., 2021) sidesteps the entire extrapolation problem by never doing that: it estimates the value function purely via expectile regression over the actions actually present in the dataset, effectively learning “how good is the best action the data happened to show us” without ever asking the Q-function to generalize beyond it.

def expectile_loss(diff, expectile=0.7):
    # Asymmetric loss: weights positive residuals (better-than-average actions) more
    weight = torch.where(diff > 0, expectile, 1 - expectile)
    return weight * diff.pow(2)

Because IQL never evaluates the Q-function on an unseen action even once, it avoids BCQ and CQL’s need for careful tuning of how conservative to be — there’s simply no extrapolation query to control.

Decision Transformer: Reframing RL as Sequence Modeling

Decision Transformer (Chen et al., 2021) drops the Bellman equation entirely. It treats a trajectory as a sequence of (return-to-go, state, action) tuples and trains an ordinary autoregressive transformer — the same architecture and loss as a language model — to predict the next action conditioned on the desired future return:

def decision_transformer_forward(model, returns_to_go, states, actions, timesteps):
    # Same next-token prediction objective as an LLM, over (R, s, a) triples instead of words
    sequence = interleave(returns_to_go, states, actions)
    embeddings = model.embed(sequence, timesteps)
    predicted_actions = model.transformer(embeddings)
    return predicted_actions

At inference time, you simply prompt the model with a high desired return, and it generates the action sequence conditioned on achieving it — turning policy optimization into supervised sequence prediction, no value function or Bellman backup required at all.

Comparison Table

Applications

• Healthcare treatment policies — learning from logged patient records where trial-and-error exploration is ethically off the table.
• Recommendation systems — training on historical logged interaction data rather than live A/B exploration against real users.
• Autonomous driving — leveraging huge logged fleet-driving datasets without needing an untrained policy to drive on real roads to collect data.
• Robotics from demonstration logs — reusing previously collected teleoperation or scripted-policy data instead of expensive new physical trials.

Key Learnings

1. Extrapolation error, not sample efficiency, is the central problem offline RL solves. Every technique here is a different strategy for preventing the Q-function (or the policy) from confidently relying on actions the data never demonstrated.
2. Conservatism has a tunable dial, and IQL’s appeal is removing the dial entirely — by never querying out-of-distribution actions, it avoids the “how conservative should CQL’s penalty be” tuning problem altogether.
3. Decision Transformer shows RL and sequence modeling aren’t as separate as they look. Once you’re willing to drop the Bellman equation, an offline RL dataset is just another sequence-prediction training set — the same insight now underlies return-conditioned and even RLHF-adjacent LLM training approaches.