Failure Analysis of Autonomous Systems with RL-Guided MCMC Sampling
Keywords: Reinforcement learning, Monte Carlo Markov chains, system validation, failure analysis
TL;DR: Reinforcement learning is used to efficiently find system failures and accurately estimate their probabilities.
Abstract: Advanced autonomous systems are increasingly deployed for critical tasks, but are typically not amenable to standard verification and validation techniques. Manually-refined Monte Carlo sampling is often the only recourse for the practical assessment of system behavior and the discovery of anomalies. However, this method scales poorly when applied to systems with high-dimensional states, multiple agents, or long time horizons. When system failures are rare, they cannot be effectively analyzed by direct sampling. We improve on previous work and demonstrate RL-MCMC, a Monte Carlo Markov chain approach to the efficient generation of rare system failures and the accurate estimation of failure mode log-likelihood. MCMC algorithms enable the sampling of arbitrary unnormalized distributions that lack an explicit generative process; however, they are highly sensitive to initialization and commonly suffer from convergence issues. We present a method to find ideal initializations for the MCMC sampling process with reinforcement learning, leveraging the power of modern neural network-based policy optimization to find solutions to nontrivial and highly-constrained sequential tasks. By formulating a Markov decision process ( MDP) to explicitly learn modal paths to failure, it is possible to eliminate the unreliable convergence phase of the MCMC algorithm and immediately begin generating valid, in-distribution system failures. We assess the approach with two simple example problems and demonstrate the accuracy and stability of the likelihood estimation.
Type Of Paper: Full paper (max page 8)
Anonymous Submission: Anonymized submission.
Submission Number: 45
Loading