More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel

Jan Grove, Michel Botros, Ylva A. Weeda, Clara I. Sánchez, Erik J Bekkers, Sybren L. Meijer, Hoel Kervadec

Keywords: Multi-rater learning, Multiple instance learning, Uncertainty estimation, Histopathology, Barrett’s Esophagus

Abstract:

In medical imaging, datasets with several expert diagnoses capture diagnostic uncertainty, yet many approaches compress diagnoses into a single consensus label.
Due to its highly subjective nature, Barret's Esophagus gradings often diverge, thus necessitating several expert opinions to mitigate variation in diagnostic or treatment outcomes. Using a multi-rater dataset from the Dutch Esophageal Pathology Panel, we propose an approach to tackle the implied issues such as poor calibration and overconfident predictions that come with a compressed label. We offer an approach that models individual rater behaviors as part of virtual panels, allowing for better prediction performance while also improving the quality of uncertainty estimates for clinical decision-making when compared to pre-compressed labels.
We show that due to their individual correlation with the clinical consensus, a combination of raters---especially an inclusion of all raters---yields higher performance and better calibrated predictions.

Primary Subject Area: Uncertainty Estimation

Secondary Subject Area: Learning with Noisy Labels and Limited Data

Registration Requirement: Yes

Reproducibility: https://github.com/jangrove379/WeakBE-Net

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 302