Towards multimodal graph neural networks for surgical instrument anticipation

Lars Wagner, Dennis N. Schneider, Leon Mayer, Alissa Jell, Carolin Müller, Alexander Lenz, Alois Knoll, Dirk Wilhelm

Abstract: Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks.