ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

Hui Wen Goh, Jonas Mueller

Keywords: active learning, data annotation, noisy labels, classification

TL;DR: ActiveLab is a straightforward method to decide which data to label next or re-label again in order to train the best model by collecting the fewest additional labels

Abstract: In real-world data labeling, annotators often provide imperfect labels. It is thus common to employ multiple annotators to label data with some overlap between their examples. We study active learning in such settings, aiming to train an accurate classifier by collecting the fewest total annotations. Here we propose ActiveLab, a practical method to decide what to label next that works with any classifier model and can be used in pool-based batch active learning with one or multiple annotators. ActiveLab automatically estimates when it is more informative to re-label examples vs. labeling entirely new ones. This is a key aspect of producing high quality labels and trained models within a limited annotation budget. In experiments on image and tabular data, ActiveLab reliably trains more accurate classifiers with far fewer annotations than a wide variety of popular active learning methods.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2301.11856/code)