Dyadic Human Relation Recognition

Kaen Kogashi, Shohei Nobuhara, Ko Nishino

Abstract: We introduce a new dataset and novel method for dyadic human relation (DHR) recognition. DHR recognition is a new image understanding task that concerns the recognition of the type (i.e., verb) and roles of a two-person interaction. Unlike past human action and dyadic interaction detection, our goal is to extract richer information regarding the roles of actors, i.e., which person is acting on whom. For this, we introduce the DHR-WebImages dataset which consists of a total of 22,046 images of 51 verb classes of DHR with per-image annotation of the verb and roles. We derive a novel model for DHR recognition by consolidating information from three streams corresponding to the visual, pose, and spatial relation features of the two people. Experimental results show that our model achieves accurate DHR recognition robust to severe occlusions and lacking visibility of actors. We believe these results will lead to richer visual information extraction.