Novel Policy Seeking with Constrained Optimization

Hao Sun, Zhenghao Peng, Bo Dai, Jian Guo, Dahua Lin, Bolei Zhou

Keywords: Novel Policy Discovery, Policy Diversity in Reinforcement Learning

Abstract:

In problem-solving, we humans tend to come up with different novel solutions to the same problem.
However, conventional reinforcement learning algorithms ignore such a feat and only aim at producing a set of monotonous policies that maximize the cumulative reward. The resulting policies usually lack diversity and novelty. In this work, we aim at enabling the learning algorithms with the capacity of solving the task with multiple solutions through a practical novel policy generation workflow that can generate a set of diverse and well-performing policies. Specifically, we begin by introducing a new metric to evaluate the difference between policies. On top of this well-defined novelty metric, we propose to rethink the novelty-seeking problem through the lens of constrained optimization, to address the dilemma between the task performance and the behavioral novelty in existing multi-objective optimization approaches, we then propose a practical novel policy seeking algorithm, Interior Policy Differentiation (IPD), which is derived from the interior point method commonly known in the constrained optimization literature. Experimental comparisons on benchmark environments show IPD can achieve a substantial improvement over previous novelty-seeking methods in terms of both the novelty of generated policies and their performances in the primal task.

One-sentence Summary: We apply novelty-gradient-free constrained optimization in diverse policy seeking tasks to generate well-performing novel policies.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/novel-policy-seeking-with-constrained/code)