Separable Tissue Representations for Attributable Risk Prediction

Victor Wåhlstrand, Jennifer Alvén, Lisa Johansson, Kristian Axelsson, Mattias Lorentzon, Ida Häggström

Abstract: Attributing model predictions to a set of variables is a crucial part of methods in medicine and the sciences. However, in multimodal settings, ablating the contribution of a particular part of an image is often challenging. We present the STRAP framework (separable tissue representations for attributable risk prediction) using a novel masked autoencoder (MAE) enabling learning representations of a varying number of image patch tokens, enhancing memory efficiency and flexibility. We apply this framework on a fracture risk prediction task using clinical features and high-resolution peripheral quantitative computed tomography (HR-pQCT) images, to investigate the contribution of bone vs. muscle and fat tissues. Unlike previous work, we are able to selectively include specific tissues in risk prediction, and attribute their contribution to the risk using ablation and state-of-the-art interpretability methods. For the first time, we demonstrate that including soft-tissue from HR-pQCT increases prediction performance both in terms of C-index and overall AUC. Source-code is openly published online: https://github.com/waahlstrand/strap.

External IDs:doi:10.1007/978-3-032-05185-1_54