Comparing Foundation Models for Medical Images: A Study on Limited Data and Generalization

Ingrid Utseth, Amund Hansen Vedal, Sarina Thomas, Line Eikvil

Keywords: foundation models, limited training data, generalization to out-of-distribution, cardiac ultrasound images

Abstract:

In this study we have investigated how vision foundation models, pretrained on different domains, compete with a specialized model for classification as a function of the size of the labeled training set of medical images. Furthermore, we have looked into the different models' ability to generalize to difficult cases.
Our experiments are conducted for cardiac ultrasound images and the downstream task of view recognition. Still, this classification task is meant to serve as a demonstrative example, where we think that the findings should be transferable to other classification tasks and other domains.
Through these experiments we found that the foundation models were able to beat the performance of our task-specific supervised model when labelled training data were limited. This was true even for models trained on natural images and when using the simple linear probing method to create a classifier. We observed that more domain-specific foundation models achieved an even higher performance with limited data. On the other hand, the more general models showed a greater ability to generalize and perform well on difficult, out-of-distribution cases. Still, for typical in-domain cases with sufficient labeled data, a task-specific ResNet model was competitive with the foundation models, while also being both smaller and faster.

Serve As Reviewer: ~Line_Eikvil1

Submission Number: 48