Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning

ACL ARR 2026 January Submission2331 Authors

Keywords: evaluation methodologies, reasoning-models evaluation

Abstract: Evaluating generative models, such as large language models (LLMs), commonly involves question-answering tasks where the final answer is selected based on probability of answer choices. On the other hand, for models requiring reasoning, the method of answer extraction plays a critical role. We reveal that the performance of reasoning models and their final answer distributions are highly sensitive to the answer extraction algorithm employed. In order to mitigate this, we propose a basic framework: Answer Regeneration. The method uses an additional model inference, providing the prior input and output prefaced by the prompt "Answer:". The final answer is then selected or extracted from the regenerated output. In the experiments, we show that this extraction-rule-agnostic approach exhibits improved performance and enhanced robustness. Furthermore, we have applied this framework to general math problems and open-ended question answering tasks. We believe our analysis and this framework could offer a more reliable results for model evaluation.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: evaluation methodologies, reasoning-models evaluation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study

Languages Studied: English

Submission Number: 2331