TAM: Temporal Adaptive Module for Video Recognition

Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, Tong Lu

Keywords: Action Recognition, Temporal Adaptive Module, Temporal Adaptive Network

Abstract: Temporal modeling is crucial for capturing spatiotemporal structure in videos for action recognition. Video data is with extremely complex dynamics along its temporal dimension due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module ({\bf TAM}) to generate video-specific kernels based on its own feature maps. TAM proposes a unique two-level adaptive modeling scheme by decoupling dynamic kernels into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a principled module and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets, demonstrate that the TAM outperforms other temporal modeling methods consistently owing to its temporal adaptive modeling strategy.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: As the video data has extremely complex dynamics along its temporal dimension, we thus propose a temporal adaptive module decoupled by a location sensitive map and a location invariant weight to capture the temporal clues in a dynamic scheme.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2005.06803/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=blTt5UqOzv