Improving NeRF Representation with No Pose Prior for Novel View Synthesis in Colonoscopy

Pedro Esteban Chavarrias-Solano, Binod Bhattarai, Sharib Ali

Abstract: Colonoscopy screening is the gold standard procedure for managing gastrointestinal malignancies. However, several limitations affect the quality of the screening process. A reliable 3D reconstruction of the surveyed area could mitigate those limitations and improve the diagnosis and treatment outcomes. Most 3D reconstruction frameworks rely on two fundamental tasks: a) a reliable camera depth prediction, and b) an accurate camera pose estimation. Although these frameworks have shown high performance in natural scenes, their impact on colonoscopy data is highly limited by the lack of annotated ground truth data. We propose a novel ray sampling technique to guide the optimisation of a neural radiance field (NeRF) system without requiring pose priors for joint estimation of camera pose prediction and novel view synthesis (NVS). Our approach is built on NoPe-NeRF [4], a method that has shown remarkable performance on NVS and camera pose estimation, but currently limited to natural scenes with a considerable amount of texture. To address this limitation, we introduce our new Depth-Weighted Patch-Focalised Random Ray Sampling (DW-PFRRS) technique and evaluate our approach on the C3VD dataset.