A Q-learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-world Environments

Shafkat Islam, Min-Hsueh Chiu, Trevor Bonjour, Ruy de Oliveira, Bharat Bhargava, Mayank Kejriwal

Abstract:

Despite substantial progress in deep reinforcement learning (DRL), a
systematic characterization of DRL agents’ robustness to unexpected events in
the environment is relatively under-studied. Such unexpected events (“novelties"),
especially those that are more structural than parametric, may significantly
deteriorate the performance of DRL agents, leading them to be unfit for
open-world environments and applications. However, not all novelties affect an
agent’s performance equally. Unfortunately, even with reasonable and constrained
definitions of the problem, the space of all novelties can be (at least) exponential.
Hence, an effective search strategy is required to find novelties that can adversely
affect the agent. This paper presents a formalism for this problem and proposes a
deep Q-learning-based novelty-search strategy that efficiently and systematically
finds candidate (potentially complex) novelties with significant negative impact on
a DRL agent. We conduct a detailed set of experiments in a stochastic multi-agent
game environment (Monopoly) with complex decision-making properties.