Policy Learning Using Weak Supervision

Jingkang Wang, Hongyi Guo, Zhaowei Zhu, Yang Liu

Keywords: Policy Learning, Noisy Reward, Imperfect Demonstration, Co-training

Abstract: Most existing policy learning solutions require the learning agents to receive high-quality supervision signals, e.g., rewards in reinforcement learning (RL) or high-quality expert demonstrations in behavioral cloning (BC). These quality supervisions are either infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the available cheap weak supervisions to perform policy learning efficiently. To handle this problem, we treat the weak supervision'' as imperfect information coming from a peer agent, and evaluate the learning agent's policy based on a correlated agreement'' with the peer agent's policy (instead of simple agreements). Our approach explicitly punishes a policy for overfitting to the weak supervision. In addition to theoretical guarantees, extensive evaluations on tasks including RL with noisy reward, BC with weak demonstrations, and standard policy co-training (RL + BC) show that our method leads to substantial performance improvements, especially when the complexity or the noise of the learning environments is high.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We propose a new way to perform policy evaluation under weak supervision.

Code: https://github.com/wangjksjtu/PeerPL

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2010.01748/code)