On Monotonicity in AI Alignment

Gilles Bareilles, Julien Fageot, Lê-Nguyên Hoang, Peva Blanchard, Wassim Bouaziz, Sébastien Rouault, Sadegh Farhadkhani, El-Mahdi El-Mhamdi

Keywords: Monotonicity, Preference Learning, Alignment, Language Models, Direct Preference Optimization, Bradley-Terry

TL;DR: We study how a reported preference for $y$ over $z$ actually impacts the valuations of $y$ and $z$ under DPO, GPO, GBT and other comparison-based preference learning methods.

Abstract: Comparison-based preference learning has become central to the alignment of AI models with human preferences. However, these methods may behave counter-intuitively. After empirically observing that, when accounting for a preference for response y over z, the model may actually decrease the probability (and reward) of generating y (an observation also made by others), this paper investigates the root causes of (non) monotonicity, for a general comparison-based preference learning framework that subsumes Direct Preference Optimization (DPO), Generalized Preference Optimization (GPO) and Generalized Bradley-Terry (GBT). Under mild assumptions, we prove that such methods still satisfy what we call local pairwise monotonicity. We also provide a bouquet of formalizations of monotonicity, and identify sufficient conditions for their guarantee, thereby providing a toolbox to evaluate how prone learning models are to monotonicity violations. These results clarify the limitations of current methods and provide guidance for developing more trustworthy preference learning algorithms.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 24188