Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

Anonymous

Keywords: keyphrase generation, keyphrase extraction, pre-trained language models

Abstract: Recent years have seen unprecedented growth in natural language understanding and generation research with the help of pre-trained language models (PLMs). Autoencoding and autoregressive language model pre-training are the two dominant techniques, and recent works unify them to excel on both natural language understanding and generation tasks. In this study, we aim to fill in the vacancy of an in-depth investigation of using PLMs for keyphrase extraction and generation. We focus on keyphrase extraction as sequence labeling and keyphrase generation as sequence-to-sequence generation. Our study investigates the performance of encoder-only versus encoder-decoder PLMs, the influence of pre-training domains, and using encoders and decoders of various depths. Experiment results on benchmarks in the scientific domain and the news domain show that (1) strong and resource-efficient keyphrase generation models can be built with in-domain encoder-only PLMs; (2) the keyphrase extraction formulation does not help the model learn to find better present keyphrases; (3) for keyphrase generation, using a deep encoder and a shallow decoder works well. Finally, we present a strong encoder-decoder model SciBART pre-trained on a large scientific corpus and demonstrate its outstanding keyphrase generation performance and advantage over state-of-the-art PLMs.

Paper Type: long

Research Area: NLP Applications