VrdONE: One-stage Video Visual Relation Detection

Xinjie Jiang, Chenxi Zheng, Xuemiao Xu, Bangzhen Liu, Weiying Zheng, Huaidong Zhang, Shengfeng He

Abstract: Video Visual Relation Detection (VidVRD) focuses on understanding how entities interact over time and space in videos, a key step for getting a deeper insight into video scenes beyond basic visual tasks. Traditional methods for VidVRD, challenged by its complexity, usually split the task into two parts: one for identifying what categories are present and another for figuring out their temporal boundaries. This split overlooks the natural connection between these elements. Addressing the need for recognizing entity independence and their interactions across a range of durations, we propose VrdONE, a streamlined yet efficacious one-stage model. VrdONE combines the features of subjects and objects, turning predicate detection into 1D instance segmentation on their combined representations. This setup allows for both category identification and binary mask generation in one go, eliminating the need for extra steps like proposal generation or post-processing. VrdONE facilitates the interaction of features across various frames, adeptly capturing both short-lived and enduring relations. Additionally, we introduce the Subject-Object Synergy (SOS) Module, enhancing how subjects and objects perceive each other before combining. VrdONE achieves state-of-the-art performances on both the VidOR benchmark and ImageNet-VidVRD, showcasing its superior capability in discerning relations across different temporal scales.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference:

Our proposed VrdONE significantly enhances multimedia processing by advancing Video Visual Relation Detection (VidVRD), crucial for in-depth video scene analysis. Its innovative one-stage approach for detecting spatial-temporal interactions among entities in videos enables more accurate and efficient multimedia applications, such as content recommendation, interactive media, and automated video editing.

In multimedia, VrdONE's impact lies in its ability to provide a nuanced understanding of video content, facilitating improved object tracking, behavior analysis, and scene interpretation. This leads to enhanced user experiences in content discovery and interaction, where accurate scene and relation detection are paramount. For example, in content recommendation systems, VrdONE can enable more precise matching of video content to user preferences by understanding the intricate dynamics within video scenes.

Additionally, VrdONE's streamlined processing model, which eliminates the need for proposal generation or post-processing, offers a more efficient method for handling large volumes of video data, crucial for content management in digital media libraries. Its effectiveness in capturing both short-term and long-term entity interactions makes it a valuable tool for developing advanced multimedia applications, including augmented reality and virtual environments, where real-time and accurate video analysis is essential.

Submission Number: 570