CONECUT: Scalable Removal of Preference Redundancy

Purbid bambroo, Daniel S. Brown, Ana Marasovic

Keywords: Reward Models, Preference Alignment, Cone Membership Test, Safety Alignment, Value Alignment, Non-Negative Least Squares (NNLS), RewardBench, Data Pruning

TL;DR: CONECUT identifies and prunes redundant preference pairs in reward model datasets, revealing inflated alignment scores on RewardBench.

Abstract: Reward models are central to post-training alignment of large language models (LLMs) via human preferences. As reward benchmarks gain prominence, it becomes critical to evaluate their integrity. A key challenge that remains underexplored in this space is the identification of redundant examples in these evaluation datasets. These are preference pairs that enforce near-duplicate or redundant half-space constraints on the reward-model weight vector and hence may inflate or exaggerate the perceived alignment of a reward model. In this work, we propose a novel method, CONECUT, to identify redundancy in preference alignment datasets by formulating this task as a cone membership test over a reward model’s hidden representations. Our experiments on RewardBench reveal that a substantial fraction of examples in evaluation pairs are near redundant, and pruning them results in measurable performance drops across multiple reward models. Our work highlights the overestimation of alignment that evaluation datasets might cause in socially critical areas like refusals and safety. We advocate for redundancy-aware evaluation as a step toward better model alignment and curating socially responsible evaluation datasets.

Submission Number: 18