Verification Limits Code LLM Training

Srishti Gureja, Elena Tommasone, Jingyi He, Sara Hooker, Matthias Gallé, Marzieh Fadaee

Keywords: LLM, code, verification, data optimization

TL;DR: We show that verification is both the bottleneck and the key in synthetic code training, and propose ways to break the verification ceiling for better generalization.

Abstract:

Large language models for code generation increasingly rely on synthetic data, where both problem solutions and verification tests are generated by models. While this enables scalable data creation, it introduces a previously unexplored bottleneck: the verification ceiling, in which the quality and diversity of training data are fundamentally constrained by the capabilities of synthetic verifiers. In this work, we systematically study how verification design and strategies influence model performance.
We investigate (i) \textbf{what} we verify by analyzing the impact of test complexity and quantity: richer test suites improve code generation capabilities (on average +3 pass@1), while quantity alone yields diminishing returns, (ii) \textbf{how} we verify by exploring relaxed pass thresholds: rigid 100\% pass criteria can be overly restrictive. By allowing for relaxed thresholds or incorporating LLM-based soft verification, we can recover valuable training data, leading to a 2–4 point improvement in pass@1 performance. However, this benefit is contingent upon the strength and diversity of the test cases used, and (iii) \textbf{why} verification remains necessary through controlled comparisons of formally correct versus incorrect solutions and human evaluation: retaining diverse correct solutions per problem yields consistent generalization gains.
Our results show that Verification as currently practiced is too rigid, filtering out valuable diversity. But it cannot be discarded, only recalibrated. By combining calibrated verification with diverse, challenging problem–solution pairs, we outline a path to break the verification ceiling and unlock stronger code generation models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 6251