Metadata Shaping: Natural Language Annotations for the Long Tail

Anonymous

Abstract: Language models (LMs) struggle to capture knowledge about rare entities. To better capture entity knowledge, a common procedure in prior work is to start with a base LM such as BERT and to modify the LM architecture or objective function to produce a knowledge-aware LM. Proposed knowledge-aware LMs perform well compared to base LMs on entity-rich tasks; however deploying, understanding, and maintaining many different specialized architectures is challenging, and they also often introduce additional computational costs. Thus we ask to what extent we can match the quality of these architectures using a base LM and only changing the data. We propose metadata shaping, a method which inserts readily available entity metadata, such as descriptions and categorical tags, into examples at train and inference time based on mutual information. Intuitively, if metadata corresponding to popular entities overlap with metadata for rare entities, the LM may be able to better reason about the rare entities using patterns learned from similar popular entities. On standard entity-rich tasks (TACRED, FewRel, OpenEntity), metadata shaping exceeds the BERT-baseline by an average of 4.3 F1 points and achieves state-of-the-art results. We further show the gains are on average 4.4x larger for the slice of examples containing tail vs. popular entities.