Investigating Design Choices for Recurrent Reasoning Models on 2D Tasks

Renee Ge, Qianli Liao, Tomaso Poggio

Keywords: 2D reasoning, recurrence, transformers

Abstract: Transformers have demonstrated remarkable performance in natural language processing and related domains, as they largely focus on sequential, autoregressive next-token prediction tasks. Yet, they struggle in logical reasoning, not necessarily because of a fundamental limitation of these models, but possibly due to the lack of exploration of more creative uses, such as latent space and recurrent reasoning. An emerging exploration in this direction is the Hierarchical Reasoning Model (Wang et al. 2025), which introduces a novel type of recurrent reasoning in the latent space of transformers, achieving remarkable performance on a wide range of 2D reasoning tasks. Despite the promising results, this line of models is still at an early stage and calls for in-depth investigation. In this work, we perform a critical review on this class of models, examine key design choices and demonstrate interesting variants that significantly outperform what is reported on the Sudoku-Extreme and Maze-Hard tasks. Our results also raise surprising observations and intriguing directions for further research.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 23685