CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
Keywords: Preference Model Pretraining
Abstract: Large language models (LLMs) have achieved notable advancements in natural language understanding and generation, driven by scalable pretraining and advanced finetuning techniques. However, improving reasoning abilities in LLMs, particularly through reinforcement learning from human feedback (RLHF), remains a challenge due to the scarcity of high-quality preference data, which is often labor-intensive to annotate and essential for reward model (RM) finetuning. To alleviate this issue, we introduce CodePMP, a scalable preference model pretraining (PMP) pipeline that leverages vast amounts of code-preference pairs synthesized from publicly available, high-quality source code. CodePMP improves the sample efficiency of RM finetuning by sufficiently pre-training preference models on synthesized code-preference pairs. In addition to validating CodePMP on widely used mathematical reasoning tasks (GSM8K, MATH), we also demonstrate its effectiveness on logical reasoning benchmarks (ReClor, LogiQA). The results consistently indicate that CodePMP significantly improves the reasoning performance of large language models (LLMs). Furthermore, our findings underscore the critical role of scalable preference model pretraining (PMP) in achieving efficient reward modeling.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4032
Loading