No Experts, No Problem: Avoidance Learning from Bad Demonstrations

Huy Hoang, Tien Anh Mai, Pradeep Varakantham

Keywords: Offline Imitation Learning, Imitation Learning, Avoidance Learning, Undesirable Demonstrations, Q Learning

TL;DR: We develop an inverse reinforcement learning approach that avoid undesirable demonstrations

Abstract:

This paper addresses the problem of learning avoidance behavior within the context of offline imitation learning. In contrast to conventional methodologies that prioritize the replication of expert or near-expert demonstrations, our work investigates a setting where expert (or desirable) data is absent, and the objective is to learn to eschew undesirable actions by leveraging demonstrations of such behavior (i.e., learning from negative examples).

To address this challenge, we propose a novel training objective grounded in the maximum entropy principle. We further characterize the fundamental properties of this objective function, reformulating the learning process as a cooperative inverse Q-learning task. Moreover, we introduce an efficient strategy for the integration of unlabeled data (i.e., data of indeterminate quality) to facilitate unbiased and practical offline training. The efficacy of our method is evaluated across standard benchmark environments, where it consistently outperforms state-of-the-art baselines.

Primary Area: Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)

Submission Number: 26371