Generating Point Cloud from Single Image in The Few Shot Scenario

Yu Lin, Jinghui Guo, Yang Gao, Yi-Fan Li, Zhuoyi Wang, Latifur Khan

Abstract: Reconstructing point clouds from images would extremely benefit many practical CV applications, such as robotics, automated vehicles, and Augmented Reality. Fueled by the advances of deep neural network, many deep learning frameworks are proposed to address this problem recently. However, these frameworks generally rely on a large amount of labeled training data (e.g., image and point cloud pairs). Although we usually have numerous 2D images, corresponding 3D shapes are insufficient in practice. In addition, most available 3D data covers only a limited amount of classes, which further restricts the models' generalization ability to novel classes. To mitigate these issues, we propose a novel few-shot single-view point cloud generation framework by considering both class-specific and class-agnostic 3D shape priors. Specifically, we abstract each class by a prototype vector that embeds class-specific shape priors. Class-agnostic shape priors are modeled by a set of learnable shape primitives that encode universal 3D shape information shared across classes. Later, we combine the input image with class-specific prototypes and class-agnostic shape primitives to guide the point cloud generation process. Experiments on the popular ModelNet and ShapeNet datasets demonstrate that our method outperforms state-of-the-art methods in the few-shot setting.