Modeling multi-scale sub-group context for group activity recognition

Keming Mao, Peiyang Jin, Yi Ping, Bo Tang

Abstract: Group activity recognition is a challenging task for complex motion and relation between actors. To utilize similar action of actors, this paper proposes a novel multi-scale Sub-group Context Block (SCB) for group Activity Recognition. Node embedding matrix and adjacent matrix are constructed and fed into SCB. In SCB, we use an assignment matrix to learn the mapping from actors to sub-groups, so the representation and interaction of sub-group can be learned automatically. Then Graph Convolution is used for further feature representation refine. In order to emphasize effect of different sub-groups, a reinforcement learning based module Sub-group Attention Block (SAB) is designed, which models it as a Markov decision process and gives each sub-group an importance value for further procedure. Multi-scale context for group activity in different levels is adopted by fusing features obtained with various clustering numbers. Finally, temporal information is integrated by multiple frames merging. Extensive experiments are performed on two standard group activity recognition datasets: the Volleyball and the Collective Activity. Our proposed method gets outstanding performance. The results also validate that SCB and SAB are effective for group activity recognition.