A Semantic-Oriented Pipeline for 3D Reconstruction of Vehicles in Urban Scenes

Matteo Frosi, Matteo Bellusci, Marco Amoruso, Matteo Matteucci

Abstract: Multiple applications require a detailed representation of the world, especially in urban scenarios, including localization, mapping, and autonomous driving. Various solutions are available to achieve the 3D reconstruction of entire urban maps, starting from point clouds, in the form of surface meshes. Nevertheless, such systems are not able to obtain precise reconstructions, which only show coarse-grained detail. To tackle this issue, while exploiting existing deep learning methods, we propose a complete pipeline for object-level 3D reconstruction, with the goal of increasing also the expressiveness of data by replacing objects' point clouds with surface meshes. While focusing only on vehicles, the method is easily extendable to other elements of the scene. We also propose a systemic approach to studying existing deep learning works on single tasks to be used in the developed pipeline. The proposed system consists of multiple steps, including: point cloud registration, semantic segmentation, clustering, object detection, point cloud completion, point cloud rendering, and 3D reconstruction. We evaluate our pipeline on sequences of the SemanticKITTI dataset, including also quantitative and qualitative analyses, which demonstrate the validity of the achieved results.