Unraveling Max-Return Sequence Modeling via Return Consistency

Zifeng Zhuang, Dengyun Peng, Jiacheng Liu, Xiao He, Xing Lei, Ting Wang, Donglin Wang

Keywords: Offline RL, Max-Return Sequence Modeling, Return Consistency

Abstract:

Offline reinforcement learning (RL) learns from fixed datasets without interaction with online environment, enabling supervised solutions for offline RL.
Decision Transformer (DT) casts offline RL as return-conditioned supervised sequence modeling, thereby sidestepping optimal value fitting and policy gradients.
This paradigm overlooks RL’s core objective of return maximization, which yields brittle behavior on suboptimal trajectories and limited stitching ability.
Reinformer reorients this objective through max-return sequence modeling: during inference, the model conditions on the predicted maximum achievable returns to generate the optimal actions.
To better understand both the SOTA performance of this paradigm and its occasional dramatic failures, we adopt a supervised perspective and introduce the return consistency to assess whether similar state-action pairs have similar returns.
Indeed, high return consistency guarantees the maximized return reliably cues the optimal action, while low consistency may lead to suboptimal action selection.
Through visualizations, two different consistency modes are exposed and we quantify this via the return standard deviation of the data cluster with highest return mean.
Furthermore, we reveal the relationship between this metric and 1) final performance, 2) context lengths, 3) model architectures through a systematic study.
Finally, we improve return consistency by explicitly decreasing the return standard deviation, thereby further increasing the performance.

Primary Area: reinforcement learning

Submission Number: 12168