Segmentation, Reconstruction and Recognition of Objects and Surfaces in 2D and 3D

Bo Sun

Abstract: Segmentation, reconstruction and recognition are classical problems in computer vision. In order to obtain an understanding of a given 2D image, we need to segment and recognize the objects through detection, instance and semantic segmentation. 2D information, however, is not enough for many computer vision applications. We need to reconstruct the 3D model of the corresponding objects in the 2D image to obtain shape information. To improve our awareness of a 3D scene, we also need the pose of each 3D object. Any 3D scene can be represented in terms of objects and surfaces, and planes are the most frequently seen type of surface, especially in man-made environments. In this dissertation, we present methods for segmentation, reconstruction and recognition in 2D and 3D data. First, we propose a method, named Oriented Point Sampling (OPS), for plane detection in 3D point clouds. Second, we present a deep learning model, dubbed Glissando-Net, to simultaneously estimate the pose and reconstruct the 3D shape of objects at the category level from a single RGB image. Last, we develop a mask refinement network for panoptic segmentation which can improve the predictions of existing panoptic segmentation methods.