Probabilistic Heatmaps and Heteroscedastic Regression for Reliable Pectoral Muscle Segmentation in Mammography

Paul Zech, Christian Huemmer, Benjamin El-Zein, Christopher Syben, Ludwig Ritschl, Steffen Kappler, Sebastian Stober

Keywords: Pectoral muscle segmentation, heteroscedastic regression, aleatoric uncertainty

TL;DR: We present a method for pectoral muscle segmentation in mammograms that models heteroscedastic uncertainty.

Abstract: Breast cancer remains a leading cause of mortality worldwide, making accurate mammography screening essential for early detection. An important preprocessing step in mammography is the accurate segmentation of the pectoral muscle as it affects downstream tasks such as breast density estimation or automated exposure control. Existing automated segmentation methods, both traditional and deep learning-based, often lack reliable confidence measures, which becomes especially problematic in the presence of occlusions or visually confounding structures such as skin folds or other muscle fibers. To address this limitation, we propose a probabilistic framework that combines heatmap-based boundary regression with heteroscedastic uncertainty to capture input-dependent variability. Our approach not only predicts the pectoral muscle boundary but also quantifies the associated uncertainty. While mainly producing unimodal predictions, the probabilistic heatmaps reveal multimodal patterns for confounding structures, further enhancing transparency in challenging cases. We demonstrate that our method provides robust and transparent means to achieve accurate segmentation while providing meaningful uncertainty estimates.

Primary Subject Area: Uncertainty Estimation

Secondary Subject Area: Segmentation

Registration Requirement: Yes

Visa & Travel: No

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 396