CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density

ICLR 2026 Conference Submission13435 Authors

Keywords: benchmark, LLM, reasoning, long-context reasoning, Cognitive Load Theory, CLT, synthetic benchmark, natural language benchmark, intrinsic difficulty, extraneous load, needle-in-a-haystack

TL;DR: CogniLoad offers a novel approach to LLM evaluation through natural language logic puzzles with independently tunable parameters (length, intrinsic difficulty, distractor density) grounded in Cognitive Load Theory

Abstract: Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT's core dimensions: intrinsic difficulty ($d$) controls intrinsic load; distractor-to-signal ratio ($\rho$) regulates extraneous load; and task length ($N$) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.

Primary Area: datasets and benchmarks

Submission Number: 13435