FeatureEndo-4DGS: Real-Time Deformable Surgical Scene Reconstruction and Segmentation with 4D Gaussian Splatting

Kai Li, Junhao Wang, William Han, Ding Zhao

Keywords: Surgical Scene Reconstruction, Feature Distillation, Gaussian Splatting

TL;DR: FeatureEndo-4DGS (FE-4DGS) is the first real-time pipeline that unifies 4D surgical scene reconstruction and semantic segmentation by distilling 2D semantic features into 4D Gaussian splatting.

Track: Proceedings

Abstract: Minimally invasive surgery (MIS) requires high-fidelity, real-time visual feedback of dynamic and low-texture surgical scenes. To address these requirements, we introduce FeatureEndo-4DGS (FE-4DGS), the first real-time pipeline leveraging feature-distilled 4D Gaussian Splatting for simultaneous reconstruction and semantic segmentation of deformable surgical environments. Unlike prior feature-distilled methods restricted to static scenes, and existing 4D approaches that lack semantic integration, FE-4DGS seamlessly leverages pre-trained 2D semantic embeddings to produce a unified 4D representation—where semantics also deform with tissue motion. This unified approach enables the generation of real-time RGB and semantic outputs through a single, parallelized rasterization process. Despite the additional complexity from feature distillation, FE-4DGS sustains real-time rendering (61 FPS) with a compact footprint, achieves state-of-the-art rendering fidelity on EndoNeRF (39.1 PSNR) and SCARED (27.3 PSNR), and delivers competitive EndoVis18 segmentation, matching or exceeding strong 2D baselines for binary segmentation tasks (0.93 DSC) and remaining competitive for multi-label segmentation (0.77 DSC).

General Area: Models and Methods

Specific Subject Areas: Medical Imaging, Foundation Models, Deployment

Data And Code Availability: Yes

Ethics Board Approval: No

Entered Conflicts: I confirm the above

Anonymity: I confirm the above

Code URL: https://github.com/kailathan/FE-4DGS

Submission Number: 183