Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction

Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, Qi Tian

Abstract:

3D skeleton-based action recognition and motion prediction are two essential problems of human activity understanding. In
many previous works: 1) they studied two tasks separately, neglecting internal correlations; 2) they did not capture sufficient relations
inside the body. To address these issues, we propose a symbiotic model to handle two tasks jointly; and we propose two scales of
graphs to explicitly capture relations among body-joints and body-parts. Together, we propose symbiotic graph neural networks, which
contains a backbone, an action-recognition head, and a motion-prediction head. Two heads are trained jointly and enhance each other.
For the backbone, we propose multi-branch multi-scale graph convolution networks to extract spatial and temporal features. The
multi-scale graph convolution networks are based on joint-scale and part-scale graphs. The joint-scale graphs contain actional graphs,
capturing action-based relations, and structural graphs, capturing physical constraints. The part-scale graphs integrate body-joints to
form specific parts, representing high-level relations. Moreover, dual bone-based graphs and networks are proposed to learn
complementary features. We conduct extensive experiments for skeleton-based action recognition and motion prediction with four
datasets, NTU-RGB+D, Kinetics, Human3.6M, and CMU Mocap. Experiments show that our symbiotic graph neural networks achieve
better performances on both tasks compared to the state-of-the-art methods. The code is relased at github.com/limaosen0/Sym-GNN