No Need to Know Everything! Efficiently Augmenting Language Models With External KnowledgeDownload PDF

Published: 18 Sept 2021, Last Modified: 05 May 2023CSKBReaders: Everyone
Keywords: Language models, knowledge bases, structured knowledge, knowledge probes
TL;DR: Allowing Language Models access to external knowledge leads to significant performance gains in knowledge intensive tasks
Abstract: Large transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture semantic, syntactic, and factual knowledge in their parameters. However, storing large amounts of factual knowledge in the parameters of the model is sub-optimal given the resource requirements and ever-growing amounts of knowledge. Instead of packing all the knowledge in the model parameters, we argue that a more efficient alternative is to provide contextually relevant structured knowledge to the model and train it to use that knowledge. This allows the training of the language model to be de-coupled from the external knowledge source and the latter can be updated without affecting the parameters of the language model. Empirical evaluation using different subsets of LAMA probe reveals that such an approach allows smaller language models with access to external knowledge to achieve significant and robust outperformance over much larger language models.
2 Replies