Learning by restoring broken 3d geometry

Jinxian Liu, Ye Chen, Bingbing Ni, Zhenbo Yu, Hang Wang

Abstract:

The key point for an experienced craftsman to repair
broken objects effectively is that he must know about them deeply.
Similarly, we believe that a model can capture rich geometry
information from a shape/scene and generate discriminative representations if it is able to find distorted parts of shapes/scenes
and restore them. Inspired by this observation, we propose a novel
self-supervised 3D learning paradigm named learning by restoring
broken shapes/scenes (collectively called 3D geometry). We first
develop a destroy-method cluster, from which we sample methods
to break some local parts of an object. Then the destroyed object
and the normal object are both sent into a point cloud network
to get representations, which are employed to segment points that
belong to distorted parts and further reconstruct/restore them to
normal. To perform better in these two associated pretext tasks,
the model is constrained to capture useful object features, such as
rich geometric and contextual information. The object representations learned by this self-supervised paradigm transfer well to
different datasets and perform well on downstream classification,
segmentation and detection tasks. Experimental results on shape
datasets and scene datasets demonstrate that our method achieves
state-of-the-art performance among unsupervised methods. We
also show experimentally that pre-training with our framework
significantly boosts the performance of supervised models.