CLEAR-WSI: Foundation Model Empowered Whole Slide Image Retrieval

Youssef Wally, Jingsong Liu, Elisabeth Wetzer, Peter J. Schüffler

Keywords: Computational Pathology, Image Retrieval, Reverse Image Search, Foundation Models, Vision Transformers, Whole Slide Images, Deep Learning

TL;DR: A fully automated, patch-count–independent reverse image search engine for H&E whole-slide images that leverages ViT foundation models, AttentionMIL aggregation, and self-review label filtering to achieve state-of-the-art retrieval and rank quality.

Abstract: The rapid growth of digital pathology has produced vast repositories of hematoxylin and eosin (H\&E) whole slide images (WSIs), yet most remain unindexed and some unlabeled, limiting their utility for computational analysis. Reverse image search provides a scalable way to organize and access these archives by retrieving visually similar slides. While current deployed retrieval systems exist, they rely on manual configuration highly affecting their performance. Thus, we propose a fully automated pathology reverse image search engine that leverages Vision Transformer (ViT) representations with attention-based multiple instance learning (AttentionMIL). The AttentionMIL framework jointly identifies diagnostically relevant WSIs and predicts slide-level diagnoses. To further improve performance, we introduce a self-reviewing classifier filtering mechanism: retrieved candidates are filtered according to their predicted labels, mostly outperforming class-informed filters. Across two public datasets, CAMELYON16 (lymph node metastases) and BRACS (breast cancer subtypes, levels 1 and 2), our method establishes new state-of-the-art results, improving $Acc_{MV}@5$ from 77.49\% to 89.92\% on CAMELYON16, from 54.12\% to 75.86\% on BRACS level-1, and from 36.47\% to 51.72\% on BRACS level-2. Beyond WSIs, our framework generalizes to WSI derived patches, enabling flexible indexing of heterogeneous H\&E image datasets. Our system is publicly available, establishing a new paradigm for content-based retrieval in computational pathology: a general-purpose, annotation-free, dataset-agnostic, and openly available search engine that scales across diverse data sources. Code: \url{https://github.com/youssefwally/FM_empowered_histopathology_IR}

Primary Subject Area: Foundation Models

Secondary Subject Area: Application: Histopathology

Registration Requirement: Yes

Reproducibility: https://github.com/youssefwally/FM_empowered_histopathology_IR

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 32