More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel

Jan Grove, Michel Botros, Ylva A. Weeda, Clara I. Sánchez, Erik J Bekkers, Sybren L. Meijer, Hoel Kervadec

Keywords: Multi-rater learning, Multiple instance learning, Uncertainty estimation, Histopathology, Barrett’s Esophagus

Abstract:

In medical imaging, datasets with several expert diagnoses capture diagnostic uncertainty, yet many approaches compress diagnoses into a single consensus label.
Due to its highly subjective nature, Barret's Esophagus gradings often diverge, thus necessitating several expert opinions to mitigate variation in diagnostic or treatment outcomes. Using a multi-rater dataset from the Dutch Esophageal Pathology Panel, we propose an approach to tackle the implied issues such as poor calibration and overconfident predictions that come with a compressed label. We offer an approach that models individual rater behaviors as part of virtual panels, allowing for better prediction performance while also improving the quality of uncertainty estimates for clinical decision-making when compared to pre-compressed labels.
We show that due to their individual correlation with the clinical consensus, a combination of raters---especially an inclusion of all raters---yields higher performance and better calibrated predictions.

Primary Subject Area: Uncertainty Estimation

Secondary Subject Area: Learning with Noisy Labels and Limited Data

Registration Requirement: Yes

Reproducibility: https://github.com/jangrove379/WeakBE-Net

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Midl Latex Submission Checklist: Ensure no LaTeX errors during compilation., Replace NNN with your OpenReview submission ID., Includes \documentclass{midl}, \jmlryear{2026}, \jmlrworkshop, \jmlrvolume, \editors, and correct \bibliography command., Did not override options of the hyperref package., Did not use the times package., Use the correct spelling and format, avoid Unicode characters, and use LaTeX equivalents instead., Any math in the title and abstract must be enclosed within $...$., Did not override the bibliography style defined in midl.cls and did not use \begin{thebibliography} directly to insert references., Avoid using \scalebox; use \resizebox when needed., Included all necessary figures and removed *unused* files in the zip archive., Removed special formatting, visual annotations, and highlights used during rebuttal., All special characters in the paper and .bib file use LaTeX commands (e.g., \'e for é)., No separate supplementary PDF uploads., Acknowledgements, references, and appendix must start after the main content.

Submission Number: 302