Dual-Phase Msqnet for Species-Specific Animal Activity Recognition

An Yu, Jeremy Varghese, Ferhat Demirkiran, Peter Buonaiuto, Xin Li, Ming-Ching Chang

Abstract: We present an effective method to detect and recognize multiple animal activities that can advance wildlife conservation and ecological research. While current activity recognition models focus on human actions, we emphasize the need for species-specific designs to accommodate the diverse and complex movements of animals. Our approach involves a dual-phase process: first identifying the species, then recognizing its activities using the cutting-edge Multi-modal Semantic Query Network (MSQNet), a Transformer-based object detector. By customizing and training our models meticulously, we address the challenges posed by the wide range of animal behaviors and physical characteristics. This highlights the importance of dedicated action recognition systems for non-human subjects. Our method achieves an impressive multi-label average precision (MAP) score of 72.5% on the Animal Kingdom dataset, demonstrating precise animal activity recognition capabilities that benefit wildlife conservation and ecological studies. This performance placed our team among the top contributions to the ICME 2024 MMVRAC challenge. Our research paves the way for real-time observation, recognition, and classification of animal behaviors in wildlife studies. Code is available at https://github.com/casperious/DP_MSQNet.