3D-SceneCaptioner: Visual Scene Captioning Network for Three-Dimensional Point Clouds

Qiang Yu, Xianbing Pan, Shiming Xiang, Chunhong Pan

Abstract: Currently, image captioning has been widely studied with the development of deep neural networks. However, seldom work has been conducted to develop captioning models for three-dimensional (3D) visual data, for example, point clouds, which are now popularly employed for vision perception. Technically, most of these models first project the 3D shapes into multiple images, and then use the existing or similar framework for image captioning models to fulfill the task. Consequently, within such a technical framework, a large amount of useful information hidden in 3D vision is inevitably lost. In this paper, a captioning model for visual scenes directly based on point clouds is proposed. First, a deep model with densely connected point convolution is developed to extract visual features directly on point clouds, and the multi-task learning method is adopted to improve the visual features. Then, the visual features are converted into sentences through a caption generation module. As a whole, an end-to-end model is constructed for the task of 3D scene captioning. This model makes full use of the rich semantic information in point clouds, and generate more accurate captions. Since there do not exist large-scale datasets for this task, in this paper two new datasets are created on existing point cloud datasets by manually labeling captions. Comprehensive experiments conducted on three datasets (including one public benchmark) indicate the effectiveness of our model.