More Room for Language: Investigating the Effect of Retrieval on Language Models

Anonymous

TL;DR: We introduce a novel theoretical framework for studying the properties of retrieval-augmented LMs, and examine how are these models different from standard models.

Abstract: Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an `ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: {i) save substantially less world knowledge in their weights, (ii) are better in understanding local context and inter-word dependencies, but {iii) are worse in comprehending global context.

Paper Type: short

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English