SeqSHAP: Subsequence Level Shapley Value Explanations for Sequential Predictions

Guanyu Jiang, Fuzhen Zhuang, Bowen Song, Yongchun Zhu, Ying Sun, Weiqiang Wang, Deqing Wang

Abstract: With the rapid development of AI techniques, models built on deep learning techniques have been applied to an increasing number of real-world scenarios. As the predictive performance of deep learning models continues to improve, their structures are also becoming increasingly complex with a large number of internal parameters, making it difficult for humans to intuitively understand their prediction mechanisms. As a result, there is an increasing demand for interpretability in machine learning (ML) applications to provide explanations for those black-box models. In recent years, many explainable AI (XAI) methods have been proposed to explain the prediction results of ML models, targeting various prediction tasks and application scenarios. However, most existing methods overlook the interpretability of sequential scenarios, which have a wide range of applications in the real world like fraud detection and sequence-based recommendation. In this paper, we propose SeqSHAP to explain sequential model predictions from a unique perspective. Compared to existing element-wise XAI methods, SeqSHAP provides intuitive explanations at the subsequence level, which explicitly models the effect of contextual information among the relevant neighboring elements in sequences. We also provide a distribution-based segmentation method to obtain reasonable subsequences from the sequential data. Extensive experiments on online transaction datasets collected from AliPay show that the proposed method could provide both faithful and reliable explanations for sequential predictions.