Squeezing performance from pathology foundation models with chained hyperparameter searches

Joseph Cappadona, Ken Gary Zeng, Carlos Fernandez-Granda, Jan Witowski, Yann LeCun, Krzysztof J. Geras

Keywords: DINO, digital pathology, hyperparameter tuning, vision transformers

TL;DR: We demonstrate that simply tuning the hyperparameters of DINOv2, using a relatively small dataset, leads to similar or superior performance than some 'state-of-the-art' models.

Abstract: Self-supervised learning (SSL) is perfectly suited for applications in digital pathology due to the scarcity of labeled data. Over the past years, many academic and industrial labs have published pathology foundation models, claiming `state-of-the-art' performance due to improvements in architecture, methodology, and/or training data. In this paper, we demonstrate that simply tuning the hyperparameters of popular SSL method DINOv2, using a relatively small dataset, leads to similar or superior performance. Specifically, we conduct three successive hyperparameter searches, iteratively increasing either dataset or model size while narrowing the hyperparameter search space and carrying over promising hyperparameters. Overall, this preliminary study demonstrates the importance of hyperparameter tuning in this domain and proposes straight-forward strategies to improve foundation models with additional compute and data.

Submission Number: 13