FLex: joint pose and dynamic radiance fields optimization for stereo endoscopic videos

Florian Philipp Stilz, Mert Asim Karaoglu, Felix Tristram, Nassir Navab, Benjamin Busam, Alexander Ladikos

Abstract:

$\textbf{Purpose}$ Reconstruction of endoscopic scenes is crucial for various medical applications, from post-surgery analysis to
educational training. However, existing methods are limited by static endoscopes, restricted deformation, or dependence on
external tracking devices for camera pose information.

$\textbf{Methods}$ We present flow-optimized local hexplanes (FLex), a novel approach addressing the challenges of a moving
stereo endoscope in a highly dynamic environment. FLex implicitly separates the scene into multiple overlapping 4D neural
radiance fields (NeRFs) and employs a progressive optimization scheme for joint reconstruction and camera pose estimation
from scratch.

$\textbf{Results}$ Tested on sequences of length up to 5000 frames, which is five times the length handled in the experiments of previous
methods, this technique enhances usability substantially. It scales highly detailed reconstruction capabilities to significantly
longer surgical videos, all without requiring external tracking information.
Conclusion Our proposed approach overcomes key limitations of existing methods by enabling accurate reconstruction and
camera pose estimation for moving stereo endoscopes in challenging surgical scenes. FLex’s advancements enhance the appli-
cability of neural rendering techniques for medical applications, paving the way for improved surgical scene understanding.
Code and data will be released on the project page: https://flexendo.github.io/