Learning Minimal Contexts: How Chain-of-Thought Induces Out-of-Distribution Generalization

Yu Wang, Fu-Chieh Chang, Pei-Yuan Wu

Keywords: large language model, chain of thought, generalization

TL;DR: OOD generalization of LLMs arises from chain-of-thought–learned minimal contexts, aided by dataset diversity and format alignment.

Abstract: Understanding how large language models (LLMs) generalize to out-of-distribution (OOD) inputs remains a central challenge, especially under strict support shifts where test data lies entirely outside the training distribution. Empirically, chain-of-thought (CoT) supervision is known to enable strong OOD generalization, yet the underlying mechanism is poorly understood. Motivated by this gap, we propose the Minimal Context Hypothesis, which posits that LLMs generalize by extracting small, task-relevant subsets of tokens that suffice for next-token prediction, rather than relying on the full input context. We study this hypothesis through tightly controlled arithmetic experiments, isolating the effects of CoT supervision, in-context learning, training distribution diversity, and input format alignment. Using train-from-scratch on a small-size language models, we combine OOD evaluation with mechanistic analyses based on saliency. Our results show that CoT supervision enables the extraction of stable Minimal Contexts, driving robust OOD generalization, which is further strengthened by unifying input format at inference time, while in-context demonstrations are not required for this effect. These findings provide actionable guidance for dataset and prompt design and suggest a general framework for understanding inductive bias and generalization in LLMs.

Submission Number: 1