Predicting the Year of Total Knee Replacement: A Transformer-Based Multimodal Approach
Keywords: Multimodal Learning, Year of TKR Prediction, Deep Learning, Knee Osteoarthritis
Abstract: Accurate prediction of the year of total knee replacement (TKR) is challenging due to the complex interplay of factors influencing the surgical decision. Current deep learning models often rely on single-modality data, limiting their predictive power. Multimodal approaches integrating imaging and patient data offer the potential to improve predictions and support clinical decisions. This study presents an end-to-end trained, transformer-based multimodal model that integrates MR imaging with tabular data, including clinical variables and image readings, to predict the year of TKR for each subject. Our model leverages cross-modal attention to fuse features from an image encoder with a self-supervised pretrained tabular encoder, achieving the highest accuracy of 63.4% among tested models. We evaluated its performance against three unimodal models and four multimodal fusion strategies, including simple concatenation, DAFT, and multimodal interaction. The results demonstrate that our model's cross-modal interaction approach with pretrained TabNet not only outperformed all unimodal models but also showed improvements over other multimodal fusion techniques, highlighting the effectiveness of cross-modal attention fusion for integrating complex data modalities in TKR year prediction tasks.
Primary Subject Area: Integration of Imaging and Clinical Data
Secondary Subject Area: Application: Radiology
Paper Type: Both
Registration Requirement: Yes
Reproducibility: https://github.com/denizlab/2025\_ISBI\_time2TKR
Visa & Travel: Yes
Submission Number: 220
Loading