Revisiting the Uniform Information Density Hypothesis in LLM Reasoning Traces

Minju Gwak, Guijin Son, Jaehyung Kim

Keywords: Information Density, s

Abstract: Chain-of-Thought reasoning improves the problem-solving abilities of large language models (LLMs), but its intermediate steps often lack faithfulness and interpretability. We propose a novel perspective on reasoning traces inspired by the Uniform Information Density (UID) hypothesis from psycholinguistics, which suggests that effective communication maintains a stable flow of information. We create step-by-step and whole-trace measures to check how evenly information is spread throughout a reasoning process. Across three challenging mathematical benchmarks, we find that unlike human communication, reasoning success correlates with low global uniformity. Meanwhile, results highlight that the UID hypothesis can be useful in developing more interpretable reasoning models.

Submission Number: 202