Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents

Hyunjun Kim

Keywords: context engineering, large language models, information theory, KL divergence, retrieval-augmented generation, multi-turn dialogue, pragmatic utility

TL;DR: An information-theoretic framework that filters LLM context by measuring whether passages shift the answer distribution toward correct responses, achieving 71.83% improvement over lexical similarity on fine-grained context selection.

Abstract: Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce **Entropic Context Shaping (ECS)**, an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures *pragmatic utility*—whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a **71.83% relative improvement** over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 52