Reinforcement Learning with Partial Order Representation for Monotonic Physical System

Yen-Jen Wang, Xiaoyu Chen, Hai Zhong, Yanjiang Guo, Jianyu Chen

Keywords: reinforcement learning, representation learning, robotics

TL;DR: We propose a training framework, which improves reinforcement learning algorithms' ability to capture the systems' monotonicity, resulting in valuable signals during training that can enhance performance and reduce sample complexity.

Abstract: Prior model-free reinforcement learning techniques may struggle with complex high-dimensional visual-motor control tasks such as rope manipulation and pouring water due to the high sample complexity involved in state representation and dynamics learning. These tasks often involve physical systems that preserve the property of monotonicity, such as water always flowing downwards when poured. Motivated by this insight, we propose the Partial Order Representation (POR) framework, which improves reinforcement learning algorithms' ability to capture the systems' monotonicity, resulting in valuable signals during training that can enhance performance and reduce sample complexity. Our experiments demonstrate that the POR framework outperforms state-of-the-art methods in terms of sample efficiency and performance across a diverse set of challenging visual-motor control tasks.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5891