Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

Josué Ruano, Martín Gómez, Eduardo Romero, Antoine Manzanera

Abstract: Highlights•A convolutional neural network, the SfSNet, estimates a pixel-wise depth map of the colon in a raw frame from the colonoscopy video.•A realistic synthetic colonoscopy database with depth annotations that is publicly available, with progressively higher levels of visual complexity.•A custom loss function which drives the network by minimizing the estimation error at folds and polyps.•Two training strategies are evaluated to optimize the depth estimation in colon folds and precancerous polyps.