Selective Preference Aggregation

Shreyas Kadekodi, Hayden McTavish, Berk Ustun

TL;DR: We propose Selective Preference Aggregation (SPA), which abstains on disputed preference pairs to produce robust, supported rankings that better align with raters.

Abstract: Many applications in machine learning and decision-making rely on procedures to aggregate human preferences.In such tasks, individual express ordinal preferences over a set of items through votes, ratings, or pairwise comparisons. We then summarize their collective preferences as a ranking. Standard methods for preference aggregation are designed to return rankings that arbitrate individual disagreements in ways that are faithful and fair. In this work, we introduce a paradigm for *selective aggregation*, where we can avoid the need to arbitrate dissent by abstaining from comparison. We summarize collective preferences as a *selective ranking* -- i.e., a partial order where we can only compare items where at least $100\cdot(1 - \tau)\%$ of individuals agree. We develop algorithms to build selective rankings that achieve all possible trade-offs between comparability and disagreement, and derive formal guarantees on their safety and stability. We conduct an extensive set of experiments on real-world datasets to benchmark our approach and demonstrate its functionality. Our results show selective aggregation can promote transparency and robustness by revealing disagreement and abstaining from arbitration.

Lay Summary:

Many decision‑making pipelines, such as grant panels, content moderation, or choosing the best model among many, combine people’s ranked opinions into one result. Standard aggregation algorithms force a total order where nearly every item is ranked independently; when voters disagree, they silently break ties.

**Selective Preference Aggregation (SPA)** produces a *tiered* list. Items are only ordered when at least  1- \$\tau\$ voters agree, while disputed pairs stay in the same tier. A lightweight graph algorithm finds the unique ranking that maximizes comparisons while respecting the chosen dissent threshold $\tau$.

Our experiments demonstrate that SPA preserves consensus information while eliminating preference inversions under adversarial noise. Our results show SPA is safe (never contradicts the majority), stable (robust to missing votes), and efficient. By signalling where consensus is strong or weak, SPA yields rankings that are transparent, robust, and harder to manipulate.

Primary Area: Social Aspects->Accountability, Transparency, and Interpretability

Keywords: Rankings, Disagreement, Preference Aggregation, Social Choice, Fairness

Submission Number: 12765