In-Context Adaptation

Yongqiang Chen, Chenxi Liu, Qingyi Guo, Bo Han, Kun Zhang

Keywords: Reasoning, In-context Learning, Transformer, LLMs, OOD generalization

Abstract: A defining characteristic of intelligence is reasoning – a capability of adapting learned knowledge to unfamiliar contexts. Although large language models (LLMs) exhibit strong reasoning capabilities in-context, it remains unclear whether they can perform human-like in-context reasoning under out-of-distribution (OOD) contexts. In this work, we introduce In-Context Adaptation (ICA), a paradigm that formalizes reasoning as the adaptive use of learned knowledge through a few demonstrations from new environments. Using a benchmark adapted from invariant learning, we show that transformers trained via next-token prediction are prone to spurious correlations and fail to reason effectively in OOD settings. To address this limitation, we propose Adaptive Context Engineering (ACE), a simple context reconstruction strategy that promotes adaptive exploitation of learned knowledge. Empirical results demonstrate significant improvements in in-context reasoning under distribution shifts, highlighting a path toward more human-like adaptive generalization in transformers.

Submission Number: 114