Temporal Relevance Analysis for Video Action Models

Quanfu Fan, Donghyun Kim, Chun-Fu Chen, Stan Sclaroff, Kate Saenko, Sarah Adel Bargal

Keywords: Temporal analysis, Frame relevance, Video data, Action recognition

TL;DR: The paper provides a deep analysis of the temporal modeling for action recognition.

Abstract:

In this paper, we provide a deep analysis of temporal modeling for action recognition, an important but underexplored problem in
the literature. We first propose a new approach to quantify the temporal relationships between frames captured by CNN-based action models based on layer-wise relevance propagation. We then conduct comprehensive experiments and in-depth analysis to provide a better understanding of how temporal modeling is affected by various factors such as dataset, network architecture, and input frames. With this, we further study some important questions for action recognition that lead to interesting findings. Our analysis shows that there is no strong correlation between temporal relevance and model performance; and action models tend to capture local temporal information, but less long-range dependencies.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2204.11929/code)