Evaluating Step-by-step Reasoning Traces: A Survey

ACL ARR 2025 February Submission5926 Authors

Abstract:

Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems.
Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, the evaluation criteria remain highly unstandardized, leading to fragmented efforts in developing metrics and meta-evaluation benchmarks. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (groundedness, validity, coherence, and utility). We then categorize metrics based on their implementations, survey which metrics are used for assessing each criterion, and explore whether evaluator models can transfer across different criteria. Finally, we identify key directions for future research.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: evaluation methodologies, evaluation, metrics, NLP datasets

Contribution Types: Surveys

Languages Studied: English

Submission Number: 5926