Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization

Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, Sungrack Yun

Keywords: Audio-Visual, Multimodal Attention, Action localization, Event localization, Weak-supervision

Abstract: Temporally localizing actions in videos is one of the key components for video understanding. Learning from weakly-labeled data is seen as a potential solution towards avoiding expensive frame-level annotations. Different from other works which only depend on visual-modality, we propose to learn richer audiovisual representation for weakly-supervised action localization. First, we propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features, which preserves the intra-modal characteristics. Second, to model both foreground and background frames, we construct an open-max classifier that treats the background class as an open-set. Third, for precise action localization, we design consistency losses to enforce temporal continuity for the action class prediction, and also help with foreground-prediction reliability. Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method effectively fuses audio and visual modalities, and achieves the state-of-the-art results for weakly-supervised action localization.

One-sentence Summary: An audio-visual fusion technique, called "multi-stage cross-attention", is developed to exploit the multi-modal representation in weakly-supervised action or event localization in untrimmed videos.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [AVE](https://paperswithcode.com/dataset/ave)