SV2MPG: From Surface Validity to Structural Validity via Two-Stage Reinforcement Learning for Olympic-Level Math Problem Generation

ACL ARR 2026 January Submission8646 Authors

Keywords: Mathematical Reasoning, Synthetic Data Generation, Reinforcement Learning, Problem Validity, Olympiad-Level Problems

Abstract:

Synthetic data has become a critical scaling strategy in modern LLM training, yet its application to high-difficulty mathematical reasoning remains limited—most methods target mid-difficulty benchmarks like GSM8K, MATH, and at most AIME, while automated generation of Olympiad-level problem–solution pairs (e.g., from AIME, HMMT, BRUMO, CMIMC) is underexplored.

To bridge this gap, we propose SV2MPG (Surface Validity and Structural Validity-aligned Mathematical Problem Generator), a lightweight generator trained via two-stage reinforcement learning to optimize both surface validity (plausibility, creativity, coherence) and structural validity (solvability, difficulty alignment). Inspired by dual-process theory, our training adopts distinct human-like evaluation perspectives: Stage 1 uses System 1 (fast, intuitive) judgments to ensure problems look right; Stage 2 uses System 2 (slow, analytical) verification to ensure they solve right.

Integrating SV2MPG with a strong open-weight solver (GPT-OSS-120B), we build an end-to-end pipeline for scalable synthesis of verified high-difficulty problems. Building on Omni-MATH, we generate SV2-MATH—a 3,707-problem dataset of Olympiad-style questions—on which supervised fine-tuning yields substantial improvements: Qwen3-0.6B and Qwen3-4B achieve +2.09% and +8.44% absolute gains in avg@4 (averaged over four 2025 Olympiad-level benchmarks), consistently outperforming models trained on multiple strong baselines, including OpenR1-Math-220k, Omni-MATH, DAPO-Math-17k, and even GPT-OSS-120B-generated data.

Paper Type: Long

Research Area: Mathematical, Symbolic, Neurosymbolic, and Logical Reasoning

Research Area Keywords: question generation, reinforcement learning, data augmentation, mathematical NLP

Contribution Types: Data resources

Languages Studied: english

Submission Number: 8646