Augmenting Open-Domain Event Detection with Synthetic Data from GPT-2

Amir Pouran Ben Veyseh, Minh Van Nguyen, Bonan Min, Thien Huu Nguyen

Abstract: Open-domain event detection (ODED) aims to identify event mentions of all possible types in text. A challenge for ODED research is the lack of large training datasets. In this work, we explore a novel method to overcome this challenge by fine-tuning the powerful pre-trained language model GPT-2 on existing datasets to automatically generate new training data for ODED. To address the noises presented in the generated data, we propose a novel teacher-student architecture where the teacher model is used to capture anchor knowledge on sentence representations and data type difference. The student model is then trained on the combination of the original and generated data and regularized to be consistent with the anchor knowledge from the teacher. We introduce novel regularization mechanism based on mutual information and optimal transport to achieve the knowledge consistency between the student and the teacher. Moreover, we propose a dynamic sample weighting technique for the generated examples based on optimal transport and data clustering. Our experiments on three benchmark datasets demonstrate the effectiveness of the propped model, yielding state-of-the-art performance for such datasets.