FAAL: Feedback-Assisted Active Learning

Anonymous

Abstract: Numerous increasingly sophisticated active learning methodologies have been introduced in recent years, each one with its own advantages. However, these methodologies have limited information available to them, as they rely only on signals derived from labeled data and model predictions. In this paper, we propose a novel approach that integrates user feedback signals into the active learning process, with the objective of enhancing the efficacy of existing methods. Our study demonstrates the consistent superiority of our approach, compared to traditional active learning methods when applied to diverse classification datasets and settings. Moreover, by incorporating user feedback via a contextual bandits algorithm, our proposed method exhibits significant additional improvements and robustness against user and annotator noise. We hope that these findings will encourage the adoption of strategies that incorporate user feedback in active learning, as well as broaden the inclusion of additional signals in the active learning process, thereby enabling maximization of limited human labeling resources. \footnote{To keep anonymity, our code will be released upon acceptance.}

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Approaches to low-resource settings, Data resources

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.