Arithmetic-Based Pretraining - Improving Numeracy Of Pretrained Language Models

Anonymous

Abstract: State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers (usually referred to as numeracy). Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both of them in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines (1) contrastive learning to improve the representation of numbers, and (2) a novel extended pretraining objective called Inferable Number Prediction Task to improve working with numbers. We evaluate our approach on three different tasks that require improved numeracy including (a) reading comprehension in the DROP dataset, (b) inference-on-tables in the InfoTabs dataset, and (c) table-to-text generation in WikiBio and SciGen datasets. Our results on DROP and InfoTabs show that our approach improves the accuracy by 9.6 and 33.9 points on these datasets, respectively. Our human evaluation on SciGen and WikiBio shows that our approach improves the factual correctness of generated outputs.

Paper Type: long

Research Area: Machine Learning for NLP