Ranking Policy Decisions

Hadrien Pouget, Hana Chockler, Youcheng Sun, Daniel Kroening

Keywords: Reinforcement Learning, Explainable AI, Statistical Fault Localization

TL;DR: We adapt Spectrum-Based Fault Localisation, a statistical method used for debugging, to ranking the relative importance of a policy's decisions.

Abstract: Policies trained via Reinforcement Learning (RL) without human intervention are often needlessly complex, making them difficult to analyse and interpret. In a run with $n$ time steps, a policy will make $n$ decisions on actions to take; we conjecture that only a small subset of these decisions delivers value over selecting a simple default action. Given a trained policy, we propose a novel black-box method based on statistical fault localisation that ranks the states of the environment according to the importance of decisions made in those states. We argue that among other things, the ranked list of states can help explain and understand the policy. As the ranking method is statistical, a direct evaluation of its quality is hard. As a proxy for quality, we use the ranking to create new, simpler policies from the original ones by pruning decisions identified as unimportant (that is, replacing them by default actions) and measuring the impact on performance. Our experimental results on a diverse set of standard benchmarks demonstrate that pruned policies can perform on a level comparable to the original policies. We show that naive approaches for ranking policies, e.g. ranking based on the frequency of visiting a state, do not result in high-performing pruned policies. To the best of our knowledge, there are no similar techniques for ranking RL policies' decisions.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/hadrien-pouget/Ranking-Policy-Decisions

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2008.13607/code)