CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming

Han Deng, Yuan Meng, SHIXIANG TANG, Wanli Ouyang, Xinzhu Ma

Keywords: Retrieval Benchmark, Embedding Model, Competitive Programming

TL;DR: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming

Abstract:

Competitive programming is widely used to evaluate the coding and reasoning abilities of large language models. However, the growing presence of duplicate or highly similar problems raises concerns not only about competition fairness, but also about the validity of competitive programming as a benchmark for model evaluation.
We introduce a retrieval-oriented benchmark suite for competitive programming, covering four retrieval tasks—two code-centric (Text-to-Code, Code-to-Code) and two newly proposed problem-centric tasks (Problem-to-Duplicate, Simplified-to-Full)—built from a combination of automatically crawled problem–solution data and manually curated annotations. Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation. We develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem–code alignment, and CPRetriever-Prob, fine-tuned for problem-level similarity. Both models achieve strong results and are open-sourced for local use. Finally, we analyze LiveCodeBench and find that high-similarity problems inflate model pass rates and reduce differentiation, underscoring the need for similarity-aware evaluation in future benchmarks.

Dataset URL: https://huggingface.co/datasets/coldchair16/CPRet-data

Code URL: https://github.com/coldchair/CPRet

Primary Area: Datasets & Benchmarks for applications in language modeling and vision language modeling

Submission Number: 1295