LFDe: A Lighter, Faster and More Data-Efficient Pre-training Framework for Event Extraction

Zhigang Kan, Liwen Peng, Yifu Gao, Ning Liu, Linbo Qiao, Dongsheng Li

Keywords: Event Extraction, Data-Efficient, Pre-training, Data generation

TL;DR: We propose a lighter, faster, and more data-efficient pre-training framework for event extraction.

Abstract: Pre-training Event Extraction (EE) models on unlabeled data is an effective strategy that frees researchers from costly and labor-intensive data annotation. However, existing pre-training methodologies necessitate substantial computational resources, requiring high-performance hardware infrastructure and extensive training duration. In response to these challenges, this paper proposes a Lighter, Faster, and more Data-efficient pre-training framework for the EE task, named LFDe. Distinct from existing methods that strive to establish a comprehensive representation space during pre-training, our framework focuses on quickly familiarizing with the task format from a small amount of automatically constructed weak-label data. It comprises three stages: weak-label data construction, pre-training, and fine-tuning. Specifically, during the weak-label data construction stage, our framework first automatically designates pseudo triggers and arguments based on the characteristics of events in real datasets to form pre-training samples. In the processes of pre-training and fine-tuning, the framework reframes event extraction as the identification of words or phrases semantically closest to the prompt within the given sentence. This paper introduces a novel prompt-based sequence labeling model for EE to accommodate this reframing. By leveraging type-aware prompt features to augment original text embeddings, it enables the conventional sequence labeling model to extract events in data-scarce scenarios. Experiments on real-world datasets show that compared to similar models, our framework requires fewer pre-training instances (only about 0.04%), a shorter pre-training period (about 0.03%), and lower memory requirements (about 57.6%). Simultaneously, our framework significantly improves performance in various data scarcity scenarios.

Track: Web Mining and Content Analysis

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 37