Investigating the Impact of Attention on Mammogram Classification

Marc Berghouse, George Bebis, Alireza Tavakkoli

Abstract: Attention, one of the most important features of modern CNNs, has been shown to improve the performance of mammogram classification, but our understanding of why attention offers improvements is rather limited. In this paper, we present the first comprehensive comparison of different combinations of baseline models and attention methods at multiple resolutions for whole mammogram image classification of masses and calcifications. Our findings indicate that attention generally helps to improve the baseline model scores, but the benefits are variable depending on the resolution and abnormality type. Furthermore, we find that pooling and overall model architecture (i.e., combination of baseline and attention) significantly impact mammogram classification scores. Specifically, scores are generally improved by architectural features that allow the model to retain as much information as possible while still focusing on relevant features. We also find that attention improves the correlation between model performance and LayerCAM activation in the region of interest. Our work provides insightful information to help guide the future construction of attention-based models for mammogram classification.