Simultaneous Detection and Interaction Reasoning for Object-Centric Action Recognition

Xunsong Li, Pengzhan Sun, Yangcen Liu, Lixin Duan, Wen Li

Abstract: The interactions between human and objects are important for recognizing object-centric actions. Existing methods usually adopt a two-stage pipeline, where object proposals are first detected using a pretrained detector, and then are fed to an action recognition model for extracting video features and learning the object relations for action recognition. However, since the action prior is unknown in the object detection stage, important objects could be easily overlooked, leading to inferior action recognition performance. In this paper, we propose an end-to-end object-centric action recognition framework that simultaneously performs Detection And Interaction Reasoning (dubbed DAIR) in one stage. Particularly, after extracting video features using a base network, we design three consecutive modules for simultaneously learning object detection and interaction reasoning. Firstly, we build a Patch-based Object Decoder (PatchDec) to generate object proposals from video patch tokens. Then, we design an Interactive Object Refining and Aggregation (IRA) to identify the interactive objects that are important for action recognition. The IRA module adjusts the interactiveness scores of proposals based on their relative position and appearance, and aggregates the object-level information into global video representation. Finally, we build an Object Relation Modeling (ORM) module to encode the object relations. These three modules together with the video feature extractor can be trained jointly in an end-to-end fashion, thus avoiding the heavy reliance on an off-the-shelf object detector, and reducing the multi-stage training burden. We conduct experiments on two datasets, Something-Else and Ikea-Assembly, to evaluate the performance of our proposed approach on conventional, compositional, and few-shot action recognition tasks. Through in-depth experimental analysis, we show the crucial role of interactive objects in learning for action recognition, and we can outperform state-of-the-art methods on both datasets. We hope our DAIR can provide a new perspective for object-centric action recognition.

External IDs:dblp:journals/tmm/LiSLDL25