Atm: Action temporality modeling for video question answering

Junwen Chen, Jie Zhu, Yu Kong

Abstract:

Despite significant progress in video question answering (VideoQA),
existing methods fall short of questions that require causal/temporal
reasoning across frames. This can be attributed to imprecise motion
representations. We introduce Action Temporality Modeling (ATM)
for temporality reasoning via three-fold uniqueness: (1) rethinking the optical flow and realizing that optical flow is effective in
capturing the long horizon temporality reasoning; (2) training the
visual-text embedding by contrastive learning in an action-centric
manner, leading to better action representations in both vision and
text modalities; and (3) preventing the model from answering the
question given the shuffled video in the fine-tuning stage, to avoid
spurious correlation between appearance and motion and hence
ensure faithful temporality reasoning. In the experiments, we show
that ATM outperforms previous approaches in terms of the accuracy on multiple VideoQAs and exhibits better true temporality
reasoning ability.