What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT

Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, Anthony Hartshorn

Keywords: Large Reasoning Models, Efficiency, Effectiveness, Chain of thought, Length, Review, Reasoning Graph, Evaluation

TL;DR: What characterizes effective reasoning? Through a comprehensive evaluation, we find that the Fraction of Failed Steps to be the strongest metric, beating length and review ratio, in both correlation and causality tests.

Abstract:

Large reasoning models (LRMs) spend substantial test-time compute on long chain-of-thought (CoT) traces, but what *characterizes* an effective CoT remains unclear. While prior work reports gains from lengthening CoTs and increasing review (revisiting earlier steps) via appended *wait* tokens, recent studies suggest that shorter thinking can outperform longer traces. We therefore conduct a systematic evaluation across ten LRMs on math and scientific reasoning. Contrary to the “longer-is-better” narrative, we find that both naive CoT lengthening and increased review are associated with *lower* accuracy.

As CoT unfolds step by step, token-level metrics can conflate verbosity with process quality. We introduce a graph view of CoT to extract structure and identify a single statistic—the *Failed-Step Fraction (FSF)*, the fraction of steps in abandoned branches—that consistently outpredicts length and review ratio for correctness across models. To probe causality, we design two interventions. First, we rank candidate CoTs by each metric at test time, where FSF yields the largest pass@1 gains; second, we edit CoTs to remove failed branches, which significantly improves accuracy, indicating that failed branches bias subsequent reasoning. Taken together, these results characterize effective CoTs as those that *fail less* and support *structure-aware* test-time scaling over indiscriminately generating long CoT.

Submission Number: 193