Semantic Scene Completion with Point Cloud Representation and Transformer-based feature fusion

Ruochong Fu, Hang Wu, Mengxiang Hao, Yubin Miao

Abstract: As a complicated computer vision task, goal of Semantic Scene Completion(s) is to predict each voxels’ occupancy and corresponding semantic category of 3D scene. In order to reduce computation burden brought by 3D convolution, recently, some methods transform voxelized scenes into point clouds through removing visible empty voxels. However, due to the inherent feature imbalance among the valid "voxel-points", reconstruction quality of these methods is limited. In this paper, we propose a novel point-based SSC to solve the dilemma. Firstly, we design a novel Surface-Attention module to compensate shortage of feature on voxels behind observed surfaces. Meanwhile, correlation between adjacent points from the same category is consolidated through Soft-Semantic Transformer layer. Experiment results on NYU and NYUCAD datasets demonstrate superiority of our method both intuitively and quantitively. Our code is available at https://github.com/furuochong.