Unsupervised Image-to-Video Domain Adaptation for Fine-Grained Video Understanding

Cristina Mata, Abe Leite, Michael S Ryoo

Keywords: Unsupervised Domain Adaptation, Image-to-video Transfer Learning, Video Segmentation

Abstract: Video understanding models continue to rely on image-pretrained semantic representations due to a lack of labeled videos. Pixel-precise video annotations are time-consuming and laborious to collect, and may not be feasibly obtained in certain situations. There is a growing amount of freely available unlabeled video data that has led many methods to tackle unsupervised video representation learning and image-to-video domain adaptation. The focus thus far has been on semantic representations for classification, which lack the spatial detail required for tasks such as segmentation. To produce representations better suited for fine-grained video understanding, we propose using large-scale image segmentation datasets and domain adversarial learning to train 2D/3D networks for video segmentation. We introduce a novel unsupervised clustered adversarial loss that first clusters feature maps from a patch embedding then applies a domain discriminator to samples within clusters. Our loss is designed to prevent removal of overall spatial structure while encouraging the removal of fine-grained spatial information specific to the image and video domains. Through experiments using several image and video segmentation datasets, we show how a general or clustered adversarial loss placed at various locations within the network can make spatial feature representations invariant to these domains and improve performance when the network has access to only labeled images and unlabeled videos.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4060