Color-based hybrid reconstruction for endoscopy

Haluk N. Tokgozoglu, Eric M. Meisner, Michael Kazhdan, Gregory D. Hager

Abstract: Three-dimensional (3D) reconstruction of images acquired during endoscopy presents an enormous opportunity for computer vision, however reconstructing geometry from such images is challenging due to lack of features. Shape From Shading (SFS) is an approach to obtain the shape of an object from a single image, but most current methods that are applicable to endoscopy are susceptible to errors caused by surfaces with differing reflectance characteristics. Another weakness of SFS is that while it has high frequency detail information, its shape is inaccurate in the low frequency sense, which makes it difficult to compare to ground truth. Multiview reconstruction (MR), on the other hand, yields reliable shape but lacks details. In this paper, we propose a novel method to perform SFS using a color projection that minimizes intensity variance caused by differing surface characteristics. We then combine the resulting reconstruction with a multiview reconstruction obtained from bundle adjustment, and combine the two reconstructions using an approach inspired by Laplacian surface editing to get a reconstruction that is accurate both in details and overall shape. We compare our results to ground truth to show improvement over existing approaches.