Fault Tolerant Reinforcement Learning via A Markov Game of Control and Stopping

David Mguni

Keywords: Fault-Tolerance, Robust Control, Reinforcement Learning, Stochastic Games, Markov Games, Optimal Stopping

TL;DR: The paper tackles fault-tolerance under random and adversarial stoppages.

Abstract:

Recently, there has been a surge in interest in safe and robust techniques within reinforcement learning (RL).
Current notions of risk in RL fail to capture the potential for systemic failures such as abrupt stoppages from system failures or surpassing of safety thresholds and the appropriate responsive controls in such instances. We propose a novel approach to fault-tolerance within RL in which the controller learns a policy can cope with adversarial attacks and random stoppages that lead to failures of the system subcomponents. The results of the paper also cover fault-tolerant (FT) control so that the controller learns to avoid states that carry risk of system failures. By demonstrating that the class of problems is represented by a variant of SGs, we prove the existence of a solution which is a unique fixed point equilibrium of the game and characterise the optimal controller behaviour. We then introduce a value function approximation algorithm that converges to the solution through simulation in unknown environments.