Systematic comparison of incomplete-supervision approaches for biomedical imaging classification

Sayedali SHETAB BOUSHEHRI, Ahmad Bin Qasim, Dominik Waibel, Fabian Schmich, Carsten Marr

Keywords: incomplete-supervision, biomedical imaging, deep learning, active learning, pre-training, transfer learning, self-supervised learning, semi-supervised learning.

TL;DR: This paper compares incomplete-supervision approaches (active learning, pre-training and semi-supervised learning) on biomedical image datasets.

Abstract:

Deep learning based image classification often requires time-consuming and expensive manual annotation by experts. Incomplete-supervision approaches including active learning, pre-training, and semi-supervised learning have thus been developed and aim to increase classification performance with a limited number of annotated images. Up to now, these approaches have been mostly benchmarked on natural image datasets, which differ fundamentally from biomedical images in terms of color, contrast, image complexity, and class imbalance.

We therefore analyzed the performance of combining seven active learning, three pre-training, and two semi-supervised methods on exemplary, fully annotated biomedical image datasets covering various imaging modalities and resolutions. For each method combination, the training started with using only 1% of labeled data. We increased the labeled training data by 5% iteratively, evaluating the performance with 4-fold cross-validation in each cycle.

The results showed that the pre-training methods ImageNet and SimCLR in combination with pseudo-labeling as the training strategy dominate the best performing combinations, while no particular active learning algorithm prevailed. For three out of four datasets, these combinations reached over 90% of the fully supervised results by only adding 25% of labeled data. An ablation study showed that pre-training and semi-supervised learning contributed up to 25% increase in macro F1-score in each cycle. In contrast, state-of-the-art active learning algorithms contributed less than 5% increase of macro F1-score in each cycle.

Based on the result of our study, we suggest employing pre-training and an appropriate incomplete-supervision training strategy for biomedical image classification when a limited number of annotated images is available. We believe that our study is an important step towards annotation-scarce and resource-efficient model training for biomedical classification challenges.

Registration: I acknowledge that publication of this at MIDL and in the proceedings requires at least one of the authors to register and present the work during the conference.

Authorship: I confirm that I am the author of this work and that it has not been submitted to another publication before.

Paper Type: validation/application paper

Primary Subject Area: Learning with Noisy Labels and Limited Data

Secondary Subject Area: Validation Study

Confidentiality And Author Instructions: I read the call for papers and author instructions. I acknowledge that exceeding the page limit and/or altering the latex template can result in desk rejection.