The Role of Active Learning in Modern Deep Learning

Thorben Werner, Lars Schmidt-Thieme, Vijaya Krishna Yalavarthi

Keywords: Deep Learning, Semi-Supervised Learning, Active Learning, Data Augmentation

TL;DR: Surprisingly, Active Learning still provides lifts, even when combined with optimized SSL and DA techniques.

Abstract:

Even though Active Learning (AL) is widely studied, it is rarely applied in contexts outside its
own scientific literature.
We posit that the reason for this is AL's high computational cost coupled with the comparatively small lifts
it is typically able to generate in scenarios with few labeled points.
In this work we study the practical setup of exhausting a fixed budget to label points from a large unlabeled
pool and designing a training pipeline to train the strongest possible model on this small labeled set.
We compare the impact of different methods to combat this low data scenario, namely data augmentation (DA),
semi-supervised learning (SSL) as options for the training pipeline and AL as selection strategy for the labeled points.
We find that AL is by far the least efficient method of solving the low data problem, generating a lift of only 1-4%
over random sampling, while DA and SSL methods can generate up to 60% lift in combination with random sampling.
However, when AL is combined with strong DA and SSL techniques, it surprisingly is still able to provide improvements.
Based on these results, we frame AL not as a method to combat missing labels, but as the final building block to
squeeze the last bits of performance out of data after appropriate DA and SSL methods as been applied.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 8693