Variable Importance on Medical Images and Socio-Demographic Data

Ahmad Chamma, Denis Alexander Engemann, Bertrand Thirion

Abstract:

Biomarker development targeting mental health is increasingly focusing on heterogeneous sources of data including brain images, biological samples and social data [1, 2, 3, 4]. Biobank initiatives give access to tens of thousands of brain images and unstructured social and biomedical data that can add context to the brain data [5, 6]. These large-scale datasets make it possible to predict biomedical outcomes using machine learning [7, 8, 9], shaping a novel prospective epidemiology framework. To interpret predictive models correctly and pave the way to causal assessments, it is crucial to understand how input features influence the prediction [10, 11, 12]. Over the past decades, a wide range of methods has been developed for ranking variables according to their importance in predictive models [13, 14, 15]. However, given the variety of settings (eg dimensionality or non-linearities, classification vs regression) it remains unclear which method provides the most accurate feature ranking for the given prediction task [16, 17]. Benchmarks have been conducted for multiple methods using simulations and empirical validation [18, 19, 20, 21], yet these efforts have been disconnected so far because of the diversity of research settings [22]. As a result, some of the most popular methods for estimating variable importance have never been systematically compared.
Here, we extend the literature by systematically comparing the most popular methods for linear and non-linear inputs in classification and regression tasks. For methods providing assessment of statistical significance, we assessed if the p-values are well calibrated. We also put performance metrics in perspective with computation time.