PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts

ICLR 2026 Conference Submission13407 Authors

Keywords: Benchmarking, Foundation Models, Multimodal Reasoning

TL;DR: We introduce PuzzleWorld, a benchmark of 667 puzzlehunt problems to test open-ended, multimodal AI reasoning, and find that models perform poorly, highlighting gaps in their creative problem-solving abilities in non-constrained environments.

Abstract: Puzzlehunts are a genre of complex, multi-step puzzles lacking well-defined problem definitions. In contrast to conventional reasoning benchmarks consisting of tasks with clear instructions and constrained environments, puzzlehunts requires discovering the underlying problem structure from multimodal evidence and iterative reasoning, mirroring real-world domains such as scientific discovery, exploratory data analysis, or investigative problem-solving. Despite progress in foundation models, their performance on open-ended settings remains largely untested. We introduce PuzzleWorld, a comprehensive benchmark of 667 puzzlehunt-style problems designed to assess step-by-step, open-ended, and creative multimodal reasoning. Each puzzle is annotated with the final solution, detailed reasoning traces, and cognitive skill labels, enabling holistic benchmarking and fine-grained diagnostic analysis. Most state-of-the-art models achieve only 1-4\% final answer accuracy. On PuzzleWorld, the best model solves only 14\% of puzzles and reaches 40\% stepwise accuracy, matching human puzzle novices but falling significantly behind puzzle enthusiasts. To demonstrate the value of our reasoning annotations, we show that fine-tuning a small model on reasoning traces boosts stepwise accuracy from 4\% to 11\%, which translates to improvements in downstream visual reasoning tasks. Our detailed error analysis reveals that current models exhibit myopic reasoning, are bottlenecked by the limitations of language-based inference, and lack sketching capabilities crucial for visual and spatial reasoning. We will publicly release PuzzleWorld to support future work on building more general, open-ended, and creative reasoning systems.

Primary Area: datasets and benchmarks

Submission Number: 13407