'Explaining RL Decisions with Trajectories’: A Reproducibility Study
Abstract: This work investigates the reproducibility of the paper "Explaining RL decisions with trajectories“ by Deshmukh et al. (2023). The original paper introduces a novel approach in explainable reinforcement learning based on the attribution decisions of an agent to specific clusters of trajectories encountered during training. We verify the main claims from the paper, which state that (i) training on less trajectories induces a lower initial state value, (ii) trajectories in a cluster present similar high-level patterns, (iii) distant trajectories influence the decision of an agent, and (iv) humans correctly identify the attributed trajectories to the decision of the agent. We recover the environments used by the authors based on the partial original code they provided for one of the environments (Grid-World), and implemented the remaining from scratch (Seaquest and HalfCheetah, Breakout, Q*Bert).
While we confirm that (i), (ii), and (iii) partially hold, we extend on the largely qualitative experiments from the authors by introducing a quantitative metric to further support (iii), and new experiments and visual results for (i). Moreover, we investigate the use of different clustering algorithms and encoder architectures to further support (ii). We could not support (iv), given the limited extent of the original experiments. We conclude that, while some of the claims can be supported, further investigations and experiments could be of interest. We recognize the novelty of the work from the authors and hope that our work paves the way for clearer and more transparent approaches.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: - Implemented the Atari Breakout environment
- Implemented the Q*Bert Breakout environment
- Added the Trajectory Transformer encoder
- Added the BERT Transformer encoder
- The paper has been heavily redacted. These new changes include, a skimmable summary of our results in Table 4, division in new subsections to avoid acronyms for the claims, moving the carbon footprint section to the appendix, and many others. Please refer to the answer to the reviewers for an exhaustive description of the changes
Assigned Action Editor: ~Aleksandra_Faust1
Submission Number: 2204
Loading