Deep Seq2Seq Keyphrase Generation: Model Calibration and Uncertainty

Anonymous

Abstract: Keyphrase generation aims to generate topical phrases from a given text either by copying from the original text (present keyphrases) or by producing new keyphrases (absent keyphrases) that capture the topical and salient aspects of the text. While many neural models have been proposed and analyzed for this task, there is limited analysis of the properties of their generative distributions at the decoding stage. Particularly, it remains to be known how well-calibrated or uncertain the confidence of different models is with empirical success rate and whether they can express their uncertainty. Here, we study the confidence scores, perplexity, and expected calibration errors of five strong keyphrase generation models with unique characteristics and designs based on seq2seq recurrent neural networks (ExHiRD), transformers with no pre-training (Transformer, Trans2Set), and transformers with pre-training (BART, and T5). We propose a novel strategy for keyphrase-level perplexity calculation and for normalizing sub-word-level perplexity to gauge model confidence.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English