LBG: LNE-based Blocking Generation Against Data Contamination on Large Language Models
Keywords: detecting data contamination, fairly evaluating LLMs
Abstract: Data contamination gradually becomes inevitable during the development of large language models (LLMs), meaning the training data commonly integrates those evaluation benchmarks unintentionally. This subsequently makes it hard to benchmark LLMs fairly. This paper introduces a novel framework called LBG (\textbf{L}NE-based \textbf{B}locking \textbf{G}eneration) for both contamination detection and mitigation for evaluating contaminated LLMs. For the first component of LBG, LBG reports a SOTA performance on our proposed length normalized entropy (LNE) to identify potential contamination by detecting anomalies on possibly contaminated LLMs. For the second component of LBG, LBG reports a SOTA performance on the mitigation of the impact of data contamination by applying LNE within a novel blocking generation strategy, specialized to adjust generation processes and re-calibrate performance metrics by suppressing the maximum value of output candidates during the generation process. We conduct extensive experiments on both contamination detection and contamination mitigation evaluation tasks, on both code generation and mathematical reasoning scenarios. The results indicate that LBG achieves an obvious SOTA performance throughout the experiments conducted in this paper. Simultaneously, LGB is lightweight and costs obviously fewer computational costs (nearly 25x) than the previous work. We hope our method will open new research avenues on data contamination for LLMs. We plan to release the resources upon publication of this work to facilitate future work.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10715
Loading