Visual Imitation Learning with Patch Rewards

Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng YAN, Zhongwen Xu

Keywords: imitation learning, reinforcement learning

TL;DR: We leverage to learn patch reward and present PatchAIL, an intuitive and principled learning framework for efficient visual imitation learning.

Abstract:

Visual imitation learning enables reinforcement learning agents to learn to behave from expert visual demonstrations such as videos or image sequences, without explicit, well-defined rewards.
Previous reseaches either adopt supervised learning techniques or induce simple and coarse scalar rewards from pixels, neglecting the dense information contained in the image demonstrations.
In this work, we propose to measure the expertise of various local regions of image samples, or called patches, and recover multi-dimensional patch rewards accordingly.
Patch reward is a more precise rewarding characterization that serves as fine-grained expertise measurement and visual explainability tool.
Specifically, we present Adversarial Imitation Learning with Patch Rewards (PatchAIL), which employs a patch-based discriminator to measure the expertise of different local parts from given images and provide patch rewards.
The patch-based knowledge is also used to regularize the aggregated reward and stabilize the training.
We evaluate our method on the standard pixel-based benchmark DeepMind Control Suite.
The experiment results have demonstrated that PatchAIL outperforms baseline methods and provides valuable interpretations for visual demonstrations.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2302.00965/code)