Establishing a Hierarchy of Training Strategies for Data-Scarce Medical Imaging

Arturas Aleksandraus, Anneke Meyer, Milda Pocevičiūtė, Daniel Jönsson, Gabriel Eilertsen, Björn Norell, Jonas Unger

Keywords: Data Scarcity, Class Imbalance, Training Strategies, Data Augmentation, Hard Adversarial Mining, Medical Image Classification, Empirical Evaluation

TL;DR: A large-scale empirical study of 3,000+ training configuration establishing a hierarchy of training strategies for data-scarce medical imaging, highlighting the synergy between augmentation and adversarial mining.

Abstract:

Deep learning in medical imaging faces fundamental constraints from data scarcity, including the inherent lack of data for rare diseases or events, and class imbalance. These challenges, compounded by privacy regulations and high expert annotation costs, make acquiring large-scale annotated datasets difficult. Although numerous training strategies aim to mitigate these issues, their comparative effectiveness in generalizing across diverse datasets remains poorly understood, providing practitioners with little guidance on prioritization.

In this paper, we investigate the effect of five training strategies: Data Augmentation, Hard Example Mining, Hard Adversarial Mining, Balancing and Reweighting, and Robustness-Oriented Training to establish a structured strategy selection for robust generalization under data scarcity.

We implement representative techniques for these families of methods and conduct 3,000+ experiments on four datasets (CIFAR-10, RetinaMNIST, OrganCMNIST, PathMNIST) with controlled scarcity.
To enable fair comparison across datasets and scarcity conditions, we introduce a Normalized Potential Score (NPS) that measures strategy effectiveness relative to the achievable improvement range, where 0.0 indicates baseline performance, 1.0 represents best achieved performance, and negative values indicate performance below baseline.

Our findings establish a hierarchy of generalization capabilities: Data Augmentation yields the largest average improvements (0.30--0.60 NPS), but still leaves a lot of performance to gain from other strategies. Combining it with Hard Adversarial Mining provides further gains (0.02--0.37 NPS). Balancing strategies enhances rare-class performance (0.08--0.10 NPS) but reduces frequent-class accuracy. We observe that Exponential Moving Average (EMA) can substantially improve training ($\pm$0.30 NPS) in some domains and has low overhead, making it a useful addition to any training pipeline. These results provide a hierarchy of strategies to consider for improving generalization in medical imaging and other data-constrained scenarios.

Primary Subject Area: Detection and Diagnosis

Secondary Subject Area: Fairness and Bias

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 24