Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMs

ACL ARR 2025 May Submission5026 Authors

Abstract: Recent advances in test-time scaling have enabled Large Language Models (LLMs) to display sophisticated reasoning abilities via extended Chain-of-Thought (CoT) generation. Despite their impressive reasoning abilities, Reasoning LLMs (RLMs) frequently display unstable behaviors, e.g., hallucinating unsupported premises, overthinking simple tasks, and displaying higher sensitivity to prompt variations. This raises a deeper research question: *How can we represent the reasoning process of RLMs to map their minds?* To address this, we propose a unified graph-based analytical framework for fine-grained modeling and quantitative analysis of RLM reasoning dynamics. Our method first clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps. Through a comprehensive analysis of derived reasoning graphs, we also reveal that key structural properties, such as exploration density, branching, and convergence ratios, strongly correlate with models' performance. The proposed framework enables quantitative evaluation of internal reasoning structure and quality beyond conventional metrics and also provides practical insights for prompt engineering and cognitive analysis of LLMs. Code and resources will be released to facilitate future research in this direction.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: reasoning large language model, interpretability, graph

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 5026