Unified Latent Steering and Residual Refinement for Online Improvement of Diffusion Policy Models

ICLR 2026 Conference Submission10917 Authors

Keywords: Offline-to-online Reinforcement Learning, Diffusion Policy, Robotic Manipulation

TL;DR: We propose USR, a unified framework that steers and refines diffusion policy models through a lightweight actor, enabling sample-efficient online adaptation for robotic manipulation.

Abstract:

Imitation learning has driven major advances in robotic manipulation by exploiting large and diverse demonstrations, yet policies trained purely by imitation remain brittle under distribution shift and novel scenarios, making online improvement essential.
Directly finetuning the parameters of modern large policies is prohibitively sample inefficient and computationally expensive,
while recent finetuning-free adaptation methods either fail to exploit the multimodal distributions learned by pretrained policies or remain confined to the coverage of demonstrations.
We propose USR, a Unified framework for latent Steering and residual Refinement that enables efficient online improvement of diffusion policy models. A lightweight actor jointly outputs latent noise to steer the diffusion process toward promising modes and residual corrections to adapt beyond the diffusion policy's support, combining stable mode selection with flexible refinement. This unified design stabilizes training and fully leverages both components. Experiments on standard benchmarks and our MultiModalBench demonstrate USR's state-of-the-art performance. Furthermore, we validate its real-world applicability by improving a Vision-Language-Action (VLA) model on a physical robot, setting a new paradigm for sample-efficient adaptation of diffusion-based policies.

Primary Area: reinforcement learning

Submission Number: 10917