When Would Vision-Proprioception Policy Fail in Robotic Manipulation?

Jingxian Lu, Wenke Xia, Yuxuan Wu, Zhiwu Lu, Di Hu

Keywords: Robotic Manipulation, Vision-Proprioception Policy

Abstract: Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported confused observations that vision-proprioception policies frequently suffer from poor generalization. In this work, we attempt to answer the question: When would vision-proprioception policy fail? To this end, we conducted temporally controlled experiments and found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy fails to take effect. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to improved generalization of vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies for robotic manipulation.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 12294