ROCA: Robust CAD Model Retrieval and Alignment from a Single Image

Can Gümeli, Angela Dai, Matthias Nießner

Abstract: We present ROCA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The code is made available at https://github.com/cangurneli/ROCA., a novel end-to-end approach that re-trieves and aligns 3D CAD models from a shape database to a single input image. This enables 3D perception of an ob-served scene from a 2D RGB observation, characterized as a lightweight, compact, clean CAD representation. Core to our approach is our differentiable alignment optimization based on dense 2D-3D object correspondences and Pro-crustes alignment. ROCA can thus provide a robust CAD alignment while simultaneously informing CAD retrieval by leveraging the 2D-3D correspondences to learn geometri-cally similar CAD models. Experiments on challenging, real-world imagery from ScanNet show that ROCA signif-icantly improves on state of the art, from 9.5% to 17.6% in retrieval-aware CAD alignment accuracy.