Discover the distinguishing and effective reasoning patterns among LLMs via an LLM

Yida Chen, Yuning Mao, Xianjun Yang, Suyu Ge, Shengjie Bi, Lijuan Liu, Saghar Hosseini, Liang Tan, Yixin Nie, Shaoliang Nie

Keywords: large reasoning models, model behavior analysis, interpretability, reasoning behaviors

TL;DR: We designed and evaluated an automatic prompt engineer algorithm that use an LLM to identify the unique and effective reasoning patterns among LLMs.

Abstract: Existing work that analyzes the reasoning behaviors of large reasoning models (LRMs) relies either on manual annotation, which is difficult to scale, or on predefined taxonomies of reasoning behaviors, which overlook the novel reasoning patterns that emerge during reinforcement learning. To address this gap, we introduce the LLM-proposed Open Taxonomy (LOT), an automatic prompt engineer algorithm that uses a language model to compare reasoning traces from LRMs and extract distinguishing patterns in their thinking processes. Our algorithm validates the quality of the LLM-extracted patterns by their accuracy in predicting the source models of unseen reasoning traces. We apply LOT to compare the reasoning of 12 open-source LRMs on tasks in math, science, and coding. LOT identifies systematic differences in their thoughts, achieving 80-100\% accuracy in distinguishing reasoning traces from LRMs that differ in scale, base model family, or objective domain. Beyond classification, LOT's natural-language taxonomy provides qualitative explanations of how LRMs think differently. Finally, in a case study, we link the reasoning differences to performance: aligning the reasoning style of smaller Qwen3 models with that of the largest Qwen3 during test time improves their accuracy on GPQA by 3.3-5.7\%.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 72