Cross-Action Cross-Subject Skeleton Action Recognition Via Simultaneous Action-Subject Learning With Two-Step Feature Removal

Yu Mitsuzumi, Akisato Kimura, Go Irie, Atsushi Nakazawa

Abstract: In this paper, we tackle a novel skeleton-based action recognition problem named Cross-Action Cross-Subject (CACS) Skeleton Action Recognition, where we can access the data of only a part of the target action classes for each training subject. Existing skeleton-based action recognition methods suffer from solving this problem because there are scarce clues to resolve the cross-entanglement of action and subject information, and the trained model will confuse those two features. To solve this challenging problem, we propose a method that consists of simultaneous action-subject learning with feature removal. In our method, 1) we use two data augmentation techniques, Bone Randomization and Phase Randomization, to roughly remove unnecessary features for respective recognitions, and then, 2) we introduce a debiased learning approach to remove the confusing features by minimizing mutual information with an action-subject-shared discriminator network. Extensive experiments on three datasets demonstrate that our method is consistently effective for several CACS problems.