SIFTA: Surgical Instrument Trajectory Forecasting with Anatomy

Gary Sarwin, Alessandro Carretta, Sara Sangalli, Victor Egon Staartjes, Matteo Zoli, Diego Mazzatenta, Luca Regli, Carlo Serra, Ender Konukoglu

Keywords: Surgical Guidance, Surgical Data Science, Anatomical Detection

TL;DR: We show that forecasting surgical instrument motion becomes feasible when anatomical context is included alongside instrument motion cues.

Abstract:

Surgical guidance can be delivered in various ways. In neurosurgery, spatial guidance and orientation are predominantly achieved through neuronavigation systems that reference pre-operative MRI scans. Recently, there has been growing interest in providing live guidance by analyzing video feeds from tools such as endoscopes. Existing approaches, including anatomy detection, orientation feedback, phase recognition, and visual question-answering, primarily focus on aiding surgeons in assessing the current surgical scene. This work explores finer scale guidance, in particular providing guidance by forecasting the trajectory of the surgical instrument, essentially trying to answer the question of what to do next?. To the best of our knowledge, this work is the first attempt to address this task for manually operated surgeries. Our approach leverages past locations of the surgical instrument as well as the relevant anatomical features.
We further account for variations in inferred camera angles. For all these, we use a specialized detection model's outputs from the same video without relying on additional input or interaction. The underlying intuition is that the anatomical context informs the surgeon’s next actions. The experiments on a comprehensive dataset containing pituitary surgery support this notion: we report an accuracy of 62.41\% by classifying the predicted direction into the four principal directions of movement, compared to 53.25\% using only the surgical instrument's historical locations. This demonstrates that anatomical features are a valuable asset in addressing this challenging task. Our findings indicate that trajectory prediction is very limited when relying only on the instrument’s past locations, but becomes noticeably more feasible when anatomical context is incorporated. This represents an important early step toward more accurate predictive models. The code will be released upon acceptance.

Primary Subject Area: Application: Endoscopy

Secondary Subject Area: Learning with Noisy Labels and Limited Data

Registration Requirement: Yes

Visa & Travel: No

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 173