Robust Counterfactual Inference in Markov Decision Processes

Jessica Lally, Milad Kazemi, Nicola Paoletti

Keywords: Counterfactual Inference, Markov Decision Processes

Abstract: This paper addresses a key limitation in existing counterfactual inference methods for Markov Decision Processes (MDPs). To make counterfactual distributions identifiable, existing approaches assume a specific causal model of the system; however, there are typically many causal models consistent with the observed and interventional distributions of an MDP, each yielding different counterfactual probabilities. Relying on a single model can therefore limit the validity and usefulness of counterfactual inference. We propose a novel \textit{non-parametric} approach that computes tight bounds on counterfactual transition probabilities across all compatible causal models. Unlike previous methods that require solving prohibitively large optimisation problems, our approach provides closed-form expressions for these bounds, making computation highly efficient even for large-scale MDPs. Using these bounds, we construct an \textit{interval} counterfactual MDP, and identify robust counterfactual policies that optimise the worst-case reward over the uncertain MDP probabilities. We evaluate our method on various case studies, demonstrating improved robustness over existing methods.

Area: Representation and Reasoning (RR)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 830