Binocular-based dense 3D reconstruction for robotic assisted minimally invasive laparoscopic surgery

Xin Sui, Yang Zhang, Xingwei Zhao, Bo Tao

Abstract: Dense 3D reconstruction of the abdominal environment for Minimally Invasive Surgery (MIS) is important for tasks in Computer Assisted Surgery (CAS), including the alignment with Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), the autonomous navigation of surgical robots, and the application of augmented reality (AR). In this paper, we investigate the binocular laparoscopy-based stereo vision technology, and ultimately achieve fast and dense 3D reconstruction of the preoperative abdominal environment and intraoperative lesion localization based on visual guidance. We introduce binocular constraints and data looping combined to improve the hand–eye calibration algorithm based on binocular laparoscopy. As it is challenging to obtain the depth truth value from medical image data, we employ a binocular unsupervised learning algorithm based on the Parallax Attention Mechanism (PAM) for depth estimation, while a coarse-to-fine pyramid optimization method is used to minimize the photometric error to obtain the laparoscopic trajectory and reconstruct the abdominal environment by parallel processing. In order to confirm the effectiveness of the algorithm, we build a binocular laparoscope-based robot platform and conduct experiments on an abdominal phantom, and the results demonstrate that the simultaneous localization and mapping (SLAM) absolute pose error (APE) of our proposed method outperforms that of some other methods, and it can achieve precise intraoperative lesion localization based on visual guidance.