What’s in a Query: Polarity-aware Distribution-based Fair Ranking

Aparna Balagopalan, Kai Wang, Olawale Elijah Salaudeen, Asia Biega, Marzyeh Ghassemi

Track: Responsible Web

Keywords: fairness, distribution-based fair ranking, query polarity

TL;DR: We propose distribution-aware metrics for measuring amortized fairness in rankings, and show the impact of query polarity in fair ranking.

Abstract: Machine learning-driven rankings, where individuals (or items) are ranked in response to a query, mediate search exposure or attention in a variety of safety-critical settings. Thus, it is important to ensure that such rankings are fair. Under the goal of equal opportunity, attention allocated to an individual on a ranking interface should be proportional to their relevance across search queries. In this work, we examine amortized fair ranking -- where relevance and attention are cumulated over a sequence of user queries to make fair ranking more feasible. Unlike prior methods that operate on expected amortized attention for each individual, we define new divergence-based measures for attention distribution-aware fairness in ranking (DistFaiR), characterizing unfairness as the divergence between the distribution of attention and relevance corresponding to an individual over time. This allows us to propose new definitions of unfairness, which are more reliable at test time and outperform prior fair ranking baselines. Second, we prove that group fairness is upper-bounded by individual fairness under this definition for a useful sub-class of divergence measures, and experimentally show that maximizing individual fairness through an integer linear programming-based optimization is often beneficial to group fairness. Lastly, we find that prior research in amortized fair ranking ignores critical information about queries, potentially leading to a fairwashing risk in practice by making rankings appear more fair than they actually are.

Submission Number: 1821