C3PO: Canonicalization of 3D Pose from Partial Views with Generalizable Correspondence Features

Yu Chi, Leonhard Sommer, Olaf Dünkel, Dominik Muhle, Daniel Cremers, Christian Theobalt, Adam Kortylewski

Keywords: Canonicalization for partial views, Correspondence feature matching

Abstract: Progress in 3D object understanding relies on the category-level canonicalization of 3D objects, i.e., bringing 3D instances into a consistent position and orientation. Most related works assume complete 3D representations, while real-world applications often require solving the more challenging task of canonicalizing from partial views, i.e., short videos that cover only a part of the object. We introduce C3PO, a method capable of canonicalizing partial views from arbitrary object categories by enforcing geometric and feature-level appearance consistency of overlapping views. We represent partial views as 3D point clouds obtained via structure-from-motion, where each point carries a feature vector that is extracted from 2D images using a novel feature extractor capable of estimating generalizable correspondence features. Notably, our correspondence features are learned on a large dataset and generalize to object categories not seen during training. On top of this, we introduce an efficient pairwise-registration framework that aligns partial object representations into a globally consistent canonical frame. Experiments on synthetic and real-world benchmarks demonstrate that C3PO significantly outperforms existing methods.

Submission Number: 333