A Q-Learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-World Environments

Shafkat Islam, Min-Hsueh Chiu, Trevor Bonjour, Ruy de Oliveira, Bharat Bhargava, Mayank Kejriwal

Abstract: Despite substantial progress in deep reinforcement learning (DRL), a systematic characterization of DRL agents’ robustness to unexpected events in the environment is relatively understudied. Such unexpected events (“novelties”), especially those that are more structural than parametric, may significantly deteriorate the performance of DRL agents, leading them to be unfit for open-world environments and applications. However, not all novelties affect an agent’s performance equally. Unfortunately, even with reasonable and constrained definitions of the problem, the space of all novelties can be (at least) exponential. Hence, an effective search strategy is required to find novelties that can adversely affect the agent. This article presents a formalism for this problem and proposes a deep Q-learning-based novelty search strategy that efficiently and systematically finds candidate (potentially complex) novelties with significant negative impact on a DRL agent. We conduct a detailed set of experiments in a stochastic multiagent game environment (Monopoly) with complex decision-making properties.

External IDs:doi:10.1109/mis.2024.3469574