EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining

Boshen Xu, Yuting Mei, liu xinbi, Sipeng Zheng, Qin Jin

Keywords: egocentric vision, video-language pretraining, multi-task learning, 3D-aware video representation

Abstract: Egocentric video-language pretraining has significantly advanced video representation learning. Humans perceive and interact with a fully 3D world, developing spatial awareness that extends beyond text-based understanding. However, most previous works learn from 1D text or 2D visual cues, such as bounding boxes, which inherently lack 3D understanding. To bridge this gap, we introduce EgoDTM, an Egocentric Depth- and Text-aware \textbf{M}odel, jointly trained through large-scale 3D-aware video pretraining and video-text contrastive learning. EgoDTM incorporates a lightweight 3D-aware decoder to efficiently learn 3D-awareness from pseudo depth maps generated by depth estimation models. To further facilitate 3D-aware video pretraining, we enrich the original brief captions with hand-object visual cues by organically combining several foundation models. Extensive experiments demonstrate EgoDTM's superior performance across diverse downstream tasks, highlighting its superior 3D-aware visual understanding. Code: \url{https://anonymous.4open.science/r/EgoDTM}.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 16359