ItinBench: Benchmarking Planning Across Multiple Cognitive Dimensions with Large Language Models

ICLR 2026 Conference Submission14903 Authors

Keywords: Natural Language Processing, Dataset and Benchmark

TL;DR: ItinBench provide a real-world level testing bed for verbal reasoning and spatial reasoning tasks for Large Language Model.

Abstract: Large language models (LLMs) with advanced cognitive capabilities are emerging as agents for various reasoning and planning tasks. Traditional evaluations often focus on specific reasoning or planning questions within controlled environments. Recent studies have explored travel planning as a medium to integrate various verbal reasoning tasks into real-world contexts. However, reasoning tasks extend beyond verbal reasoning alone, and a comprehensive evaluation of LLMs requires a testbed that incorporates tasks from multiple cognitive domains. To address this gap, we introduce ItinBench, a benchmark that features a distinct cognitive dimension, i.e., spatial reasoning, into trip itinerary planning while keeping the traditional verbal reasoning tasks. ItinBench evaluates various LLMs across diverse tasks simultaneously, including Llama 3.1 8B, Mistral Large, Gemini 1.5 Pro, and GPT family. Our findings reveal that LLMs struggle to maintain high and consistent performance when concurrently handling multiple cognitive dimensions. By incorporating tasks from distinct human-level cognitive domains, ItinBench provides new insights into building more comprehensive reasoning testbeds that better reflect real-world challenges. The code and dataset are attached.

Primary Area: datasets and benchmarks

Submission Number: 14903