A multivariate Markov chain model for interpretable dense action anticipation

Yihui Qiu, Deepu Rajan

Abstract: Dense action anticipation, as opposed to next action anticipation, deals with predicting multiple actions over a long horizon of a few minutes. Recent approaches for dense action anticipation are based on deep learning, which lacks interpretability in that the models cannot explain the decisions made through a causal relationship between past observations and predictions. In this paper, we propose a Goal oriented multivariate Markov chain (GoMMC) for interpretable dense action anticipation, which can capture the influence between various objects and their interactions, allowing for a probabilistic selection of actions performed in the long term. Experiments on 50Salads and Breakfast datasets show that the proposed model performs better than deep learning models when ground truth information on past objects and actions is available.