New News: System-2 Fine-tuning for Robust Integration of New Knowledge

ICLR 2026 Conference Submission21970 Authors

Keywords: Fine-tuning, Belief Update, Continual Learning, In-Context Learning

TL;DR: This paper presents "New News",a dataset evaluating how language models internalize new information, and shows that System-2 Fine-tuning (Sys2-FT) with self-QA enhances knowledge consolidation.

Abstract: Humans and intelligent animals can internalize new information and accurately internalize their implications to perform downstream tasks. While large language models (LLMs) can achieve this through in-context learning (ICL) when the information (news) is explicitly given as context, adequately integrating the information into model weights via fine-tuning remains challenging. In this paper, we introduce New News, a dataset composed of hypothetical yet plausible news spanning multiple domains (mathematics, coding, discoveries, leaderboards, events), accompanied by downstream evaluation questions whose correct answers critically depend on understanding and internalizing the news. First, we demonstrate a substantial gap between naive fine-tuning and in-context learning (FT-ICL gap) on our dataset. To address this gap, we explore a suite of self-play data generation protocols — paraphrases, implications, and Self-QA — designed to distill the knowledge processed by the model with context into the weights of the model, which we term System-2 Fine-tuning (Sys2-FT). We systematically evaluate ICL and Sys2-FT performance across data domains and model scales with the Qwen 2.5 family of models. Our results demonstrate that the Self-QA protocol of Sys2-FT significantly improves models’ in-weight learning of the news while preserving general capabilities. Furthermore, we discover the contextual shadowing effect, where training with the news in context followed by its rephrases or QAs catastrophically degrades learning of the news. Finally, we show preliminary evidence of an emerging scaling law of Sys2-FT.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 21970