Pre-training Limited Memory Language Models with Internal and External Knowledge

Linxi Zhao, Sofian Zalouk, Christian Belardi, Justin Lovelace, Jin Peng Zhou, Kilian Q Weinberger, Yoav Artzi, Jennifer J. Sun

Keywords: Knowledge Separation, Pretrained Large Language Models, Reliable, editable and verifiable Language Models

TL;DR: A new class of language models that offloads factual knowledge to an external database rather than encoding it in their parameters

Abstract: Neural language models are black boxes—both linguistic patterns and factual knowledge are distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We propose a new class of language models, Limited Memory Language Models (LMLM), with a pre-training recipe that stores factual knowledge in both internal weights and an external database. Our approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model to perform targeted lookups rather than relying on memorization in model weights. Our experiments demonstrate that LMLMs achieve competitive performance compared to significantly larger, knowledge-dense LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases. This work represents a fundamental shift in how language models interact with and manage factual knowledge.

Serve As Reviewer: ~Linxi_Zhao1

Submission Number: 29