Knowledgeless Language Models: Decoupling Linguistic Competence and Factual Knowledge

ICLR 2026 Conference Submission14541 Authors

Keywords: Factuality, Hallucination, Pretraining, Bias

TL;DR: Language models can be trained not to memorize factual knowledge while maintaining strong linguistic and reasoning capabilities

Abstract: Language models capture a broad spectrum of human knowledge due to being trained on large and diverse real-world datasets. However, this knowledge is not always necessary for linguistic tasks and can contribute to hallucinated outputs, as real-world knowledge is inherently dynamic and context-dependent. Such behavior limits their applicability in domains where factual precision is critical, such as healthcare and law. Moreover, LLMs trained on large text corpora inevitably inherit societal biases present in their sources. In this work, we introduce Knowledgeless LMs (KLLMs), a class of models intentionally pretrained to forgo memorization of entity-specific knowledge while retaining structural and semantic understanding of language. We present our approach for designing and training these models and evaluate them across a spectrum of downstream tasks, including language understanding, commonsense reasoning, and context-based factual benchmarks. Our results show that KLLMs achieve competitive or superior performance compared to fully parametric LLMs, particularly when provided with the relevant context, while substantially reducing reliance on memorized world knowledge. This leads to lower hallucination risks and improved calibration, with more reliable confidence estimates. Overall, KLLMs demonstrate that strong linguistic and reasoning capabilities can be maintained without extensive factual memorization, highlighting knowledgeless pretraining as a promising paradigm for building more efficient, faithful, and controllable language models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14541