Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis

Jiayu Fu, Mourad Heddaya, Chenhao Tan

Track: long paper (up to 10 pages)

Keywords: Benchmark Generation, Hypotheses Generation, LLM, LLM Evaluation

TL;DR: Proposing a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses.

Abstract: Numerous math benchmarks exist to evaluate LLMs' mathematical capabilities. However, most involve extensive manual effort and are difficult to scale. Consequently, they cannot keep pace with LLM development or easily provide new instances to mitigate overfitting. Some researchers have proposed automatic benchmark generation methods, but none focus on identifying the specific math concepts and skills on which LLMs are error-prone, and most can only generate category-specific benchmarks. To address these limitations, we propose a new math benchmark generation pipeline that uses AI-generated hypotheses to identify the specific math concepts and skills that LLMs struggle with, and then generates new benchmark problems targeting these weaknesses. Experiments show that hypothesis accuracy positively correlates with the difficulty of the generated problems: problems generated from the most accurate hypotheses reduce Llama-3.3-70B-Instruct's accuracy to as low as 45\%, compared to 77\% on the original MATH benchmark. Furthermore, our pipeline is highly adaptable and can be applied beyond math to explore a wide range of LLM capabilities, making it a valuable tool for investigating how LLMs perform across different domains.

Presenter: ~Jiayu_Fu1

Format: No, the presenting author is unable to, or unlikely to be able to, attend in person.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Funding: No, the presenting author of this submission does *not* fall under ICLR’s funding aims, or has sufficient alternate funding.

Submission Number: 91