When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation

Anonymous

Abstract: Data Augmentation (DA) is vital in deep learning to improve the generalizability of neural networks. Most of existing DA techniques in NLP naively add a certain number of augmented samples without paying attention to the quality and added computational cost of these samples. Furthermore, state-of-the-art DA techniques in the literature usually learn to generate or re-weight augmented samples more specific to the main task; however, these learning-based DA techniques are not sample-efficient and they are computationally expensive. In this work, we propose a universal DA technique, called Glitter, for NLP which aims at efficiency and performance at the same time. In other words, Glitter can be applied to any existing DA technique to improve its training efficiency and sample efficiency and maintain its competitive performance. We evaluate Glitter on several downstream tasks such as the GLUE benchmark, SQuAD, and HellaSwag in a variety of scenarios including general single network, consistency training, self-distillation and knowledge distillation (KD) setups.