MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Keywords: LLM Refinement, Reasoning, Multi-Agent
Abstract: Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples for each problem and aggregating over them to find a better answer. While these improve performance, they often reach a saturation point beyond which additional samples provide no return. Refinement offers an alternative by using model-generated feedback to improve answer quality. However, refinement introduces three key challenges: (1) Excessive refinement: Uniformly refining all instances can cause over-correction and reduce overall performance. (2) Inability to localize and address errors: LLMs have a limited ability to self-correct and struggle to identify and correct their own mistakes in a targeted way. (3) Insufficient refinement: Deciding how many iterations of refinement are needed is non-trivial, and stopping too soon could leave errors unaddressed. To tackle these issues, we propose MAgICoRe, a framework for Multi-Agent Iteration for Coarse-to-fine Refinement. MAgICoRe aims to avoid excessive refinement by categorizing problems as easy or hard, solving easy problems with coarse-grained aggregation, and solving hard ones with fine-grained and iterative multi-agent refinement. To enable more granular error localization, we incorporate external step-wise reward model (RM) scores. To ensure effective refinement, we employ a multi-agent loop with three agents: the Solver, the Reviewer (which generates targeted feedback based on step-wise RM scores) and the Refiner (which incorporates feedback and generates new solutions). To ensure sufficient refinement, we re-evaluate updated solutions, iteratively initiating further rounds of multi-agent refinement. We evaluate MAgICoRe on Llama-3-8B and GPT-3.5 and show its effectiveness across five math reasoning datasets, with consistent gains for all datasets and models. Even one iteration of MAgICoRe beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% while using less than 50% of the samples. Unlike iterative refinement with baseline methods, MAgICoRe continues to improve with more iterations. Finally, our ablations highlight the importance of MAgICoRe's use of RMs and multi-agent communication.
Supplementary Material: zip
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 11411
Loading