Unsupervised Feature Extraction from a Foundation Model Zoo for Cell Similarity Search in Oncological Microscopy Across Devices

Gabriel Kalweit, Anusha Klett, Mehdi Naouar, Jens Rahnfeld, Yannick Vogt, Diana Laura Infante Ramirez, Rebecca Berger, Jesus Duque Afonso, Tanja Nicole Hartmann, Marie Follo, Michael Luebbert, Roland Mertelsmann, Evelyn Ullrich, Joschka Boedecker, Maria Kalweit

Keywords: Foundation Models, Nearest Neighbor Search, Cell Imaging

Abstract: Acquiring high-quality datasets in medical and biological research is costly and labor-intensive. Traditional supervised learning requires extensive labeled data and faces challenges due to diverse imaging equipment and protocols. We propose Entropy-guided Weighted Combinational FAISS (EWC-FAISS), using foundation models trained on natural images without fine-tuning, as feature extractors in an efficient and adaptive k-nearest neighbor search. Our approach shows superior generalization across diverse conditions, achieving competitive performance compared to fine-tuned DINO-based models and NMTune, whilst reducing computational demands. Experiments validate the effectiveness of EWC-FAISS for efficient and robust cell image analysis.

Submission Number: 86