MIXCON3D: SYNERGIZING MULTI-VIEW AND CROSS-MODAL CONTRASTIVE LEARNING FOR ENHANCING 3D REPRESENTATION
Keywords: 3D open-world understanding, cross-model pre-training, contrastive learning
Abstract: By adapting the success of Contrastive Language-Image Pre-training from 2D vision to the 3D world, contrastive learning has emerged as a promising paradigm for open-world understanding jointly with text, image, and point cloud. While existing studies focus on aligning features from these individual modalities, this paper introduces a novel joint representation alignment approach. This mechanism enriches the conventional tri-modal representation by creating a new combined representation of images and point clouds, thus offering more accurate depiction of real-world 3D objects and bolstering text alignment. The method, termed as MixCon3D, is furthered through the integration of multi-view images, offering a more holistic representation. Furthermore, we pioneer the first thorough investigation of various training recipes for the 3D contrastive learning paradigm, building a strong baseline with improved performance and generalizability. Extensive experiments conducted on three representative benchmarks reveals that our method renders significant improvement over the baseline, surpassing the previous stateof-the-art performance on the challenging 1,156-category Objaverse-LVIS dataset by 5.7%. We further showcase the effectiveness of our approach in more applications, including text-to-3D retrieval and point cloud captioning.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4974
Loading