MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks
Abstract: Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets.
We observe that while LLMs are not designed for user-specified information types, they have a decent sense of \emph{important information}, i.e., the meta-understanding of IE.
Therefore, we propose a novel framework MetaIE to build a small LM as a meta-model by learning to extract "important information", such that this meta-model can be effectively and efficiently adapted to all kinds of (few-shot) IE tasks.
Specifically, we obtain the small LM via a symbolic distillation from an LLM.
We construct the distillation dataset via sampling sentences from language model pre-training datasets and prompting an LLM to identify the typed spans of ``important information''.
Extensive results on 13 datasets from 6 IE tasks
confirm that MetaIE can offer a better starting point for few-shot adaptation
and outperform other strong meta-models,
including a multi-task model built upon multiple large IE benchmark training sets.
Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: Information Extraction, Meta-Learning, Transfer Learning
Contribution Types: Approaches to low-resource settings
Languages Studied: English
Submission Number: 1022
Loading