Multi-view 3D Objects Localization from Street-Level Scenes

Javed Ahmad, Matteo Toso, Matteo Taiana, Stuart James, Alessio Del Bue

Abstract: This paper presents a method to localize street-level objects in 3D from images of an urban area. Our method processes 3D sparse point clouds reconstructed from multi-view images and leverages 2D instance segmentation to find all objects within the scene and to generate for each object the corresponding cluster of 3D points and matched 2D detections. The proposed approach is robust to changes in image sizes, viewpoint changes, and changes in the object’s appearance across different views. We validate our approach on challenging street-level crowd-sourced images from the Mapillary platform, showing a significant improvement in the mean average precision of object localization for the available Mapillary annotations. These results showcase our method’s effectiveness in localizing objects in 3D, which could potentially be used in applications such as high-definition map generation of urban environments. The code is publicly available ( https://github.com/IIT-PAVIS/Multi-view-3D-Objects-Localization-from-Street-level-Scenes ).