GaussianClin: Multimodal Featured Gaussian Splatting for Dynamic Clinical Videos
Keywords: 3D Scene Reconstruction, Robot-assisted Clinical Application
TL;DR: Our novel dynamic Gaussian Splatting method for surgical videos integrates foundation features and cross-frame relations, enhancing medical understanding and tissue deformation handling.
Abstract: Reconstructing dynamic 3D models from clinical videos is crucial for medical applications such as surgical visualization, robot-assisted surgery, and medical training. However, the clinical environment presents unique challenges, including limited surface textures, inconsistent lighting, and the need for expert-level medical knowledge, making it difficult for non-experts to directly apply existing techniques. To address these challenges, we present **GaussianClin**, a novel approach that enhances 3D modeling capabilities in dynamic clinical videos by leveraging multimodal feature-based Gaussian splatting (GS). By embedding trained multimodal feature fields into the radiance field, **GaussianClin** integrates general medical knowledge and improves the performance of GS in tasks like 3D tissue visualization, real-time object enhancement, clinical instrument and organ segmentation, and medical visual question answering. To effectively capture temporal dynamics and tissue deformations, we further introduce a spatiotemporal graph distillation, which significantly improves handling deformable tissues compared to standard GS methods. Experimental results demonstrate that **GaussianClin** enables clinical 3D expert models to leverage massive pre-trained 2D multimodal foundation models, thereby paving the way for advancements in robot-assisted surgery and medical data processing.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2620
Loading