Improving LLM-Generated Code Quality with GRPO

Maxime Robeyns, Laurence Aitchison

Keywords: LLM, Codegen, RL

TL;DR: We train LLMs with RL for code generation, going beyond code correctness rewards and focusing on notions of quality, maintainability and security

Abstract: Large Language Models (LLMs) are gaining widespread use for code generation. Recent training procedures use execution feedback as a reward signal, typically focusing on the functional correctness of the code, using unit test pass rate as a reward signal. However, this reward signal fails to capture notions of maintainability, quality and safety of the code produced. We address this under-explored area and develop a comprehensive library to quantify various aspects of code quality, and use it as a reward in GRPO. We find GRPO increases code quality according to this measure, which is confirmed by expert, blinded human annotators.

Submission Number: 6