3D-COCO: Extension of MS-COCO Dataset for Scene Understanding and 3D Reconstruction

Bideaux Maxence, Phe Alice, Mohamed Chaouch, Luvison Bertrand, Quoc-Cuong Pham

Abstract: We introduce 3D-COCO, an extension of the original MS-COCO [1] dataset providing 3D models and 2D-3D alignment annotations. 3D-COCO was designed to achieve computer vision tasks such as 3D reconstruction or image detection configurable with textual, 2D image, and 3D CAD model queries. We complete the existing MS-COCO [1] dataset with 28 K 3D models collected on ShapeNet [2] and Objaverse [3]. By using an IoU-based method, we match each MS-COCO [1] annotation with the best 3D models to provide a 2D-3D alignment. The open-source nature of $3 \mathrm{D}-\mathrm{COCO}$ is a premiere that should pave the way for new research on 3D-related topics. The dataset and its source codes is available at https://kalisteo.cea.fr/index.php/ coco3d-object-detection-and-reconstruction/