Multi-Modal Action Recognizer Bridges Human Motion Generation and Understanding

Jidong Kuang, Hongsong Wang, Jie Gui

Keywords: Skeleton-Based Human Action Recognition, Motion Generation

TL;DR: We introduce a unified framework that establishes a bidirectional connection between human motion and language semantics.

Abstract: Human action recognition and motion generation are two active research problems in human-centric computer vision, both aiming to align motion with textual semantics. However, most existing works study these two problems separately, without uncovering the bidirectional links between them, namely that motion generation requires semantic comprehension. This work investigates unified action recognition and motion generation by leveraging skeleton coordinates for both motion understanding and generation. We propose Coordinates-based Autoregressive Motion Diffusion (CoAMD), which synthesizes motion in a coarse-to-fine manner. As a core component of CoAMD, we design a Multi-modal Action Recognizer (MAR) that provides semantic guidance for motion generation. Our model can be applied to four important tasks, including skeleton-based action recognition, text-to-motion generation, text–motion retrieval, and motion editing. Extensive experiments on 13 benchmarks across these tasks demonstrate that our approach achieves state-of-the-art performance, highlighting its effectiveness and versatility for human motion modeling.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 11942