Realistic Surgical Simulation from Monocular Videos

Kailing Wang, Chen Yang, Keyang Zhao, Xiaokang Yang, Wei Shen

Keywords: Surgical Simulation, Video-based Reconstruction, Robotic Surgery

TL;DR: We introduce an automatic system that reconstructs geomery consistent simulation scenes from monocular videos and perform simulation with Visco-Elastic physics model.

Abstract: This paper tackles the challenge of automatically constructing realistic surgical simulation systems from readily available surgical videos. Recent efforts have successfully integrated physically grounded dynamics within 3D Gaussians to perform high-fidelity simulations in well-reconstructed static simulation environments. However, they struggle with the geometry inconsistency of simulation environments and unrealistic physical deformations of soft tissues when it comes to dynamic and complex surgical processes. In this paper, we propose SurgiSim, a novel automatic simulation system to overcome these limitations. To build a surgical simulation environment, we maintain a canonical 3D scene composed of 3D Gaussians coupled with a deformation field to accurately model monocular dynamic surgical scenes. This process involves a multi-stage optimization with trajectory and anisotropic regularization, enhancing the geometry consistency of the canonical scene which serves the simulation environment. To improve the realism of physical simulations, we implement a Visco-Elastic deformation model based on the Maxwell model, effectively restoring the complex deformations of tissues. Additionally, we estimate the physical properties of tissues by minimizing the discrepancies between the input video and simulation results guided by predicted tissue motion, ensuring realistic simulation outcomes. Experiments across diverse surgical scenarios demonstrate SurgiSim's ability to perform realistic physical interactions of soft tissues among surgical procedures, showing its enormous potential for enhancing surgical training, planning, and robotic surgery systems.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2800