Measuring Iterative Temporal Reasoning with Time Puzzles

ACL ARR 2026 January Submission3137 Authors

Keywords: temporal reasoning, tool augmented reasoning, iterative reasoning

Abstract: We introduce Time Puzzles, a constraint-based date inference task for evaluating iterative temporal reasoning. Each puzzle combines factual temporal anchors with (cross-cultural) calendar relations, admits one or multiple valid solution dates, and is algorithmically generated for controlled, dynamic, and continual evaluation. Across 13 diverse LLMs, Time Puzzles well distinguishes their iterative temporal reasoning capabilities and remains challenging without tools: GPT-5 reaches only 49.3% accuracy and all other models stay below 31%, despite the dataset's simplicity. Web search consistently yields substantial gains and using code interpreter shows mixed effects, but all models perform much better when constraints are rewritten with explicit dates, revealing a gap in reliable tool use. Overall, TimePuzzles presents a simple, cost-effective diagnostic for tool-augmented iterative temporal reasoning.

Paper Type: Short

Research Area: Resources and Evaluation

Research Area Keywords: logical reasoning, benchmarking, NLP datasets, evaluation

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: English

Submission Number: 3137