Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

Gholamali Aminian, Idan Shenfeld, Amir R. Asadi, Ahmad Beirami, Youssef Mroueh

Keywords: Best-of-N (BON) sampling, Alignment, KL divergence, Regret

TL;DR: We study Best-of-N when there is a discrepancy between the proxy reward used at inference time and the true reward. We show that the recently introduced soft BON algorithm can help mitigate reward overoptimization both theoretically and empirically.

Abstract:

A simple yet effective method for inference-time alignment of generative models is Best-of-$N$ (BoN), where $N$ outcomes are sampled from a reference policy, evaluated using a proxy reward model, and the highest-scoring one is selected.
While prior work argues that BoN is almost optimal in reward vs KL tradeoffs, the effectiveness of BoN depends critically on the quality of the proxy reward model used for selection. For this purpose, we study BoN through a smooth version known as Soft Best-of-N (SBoN) and develop a theoretical framework to address this gap. We analyze the scaling behaviour of BoN by providing bounds on the KL divergence between the SBoN policy and the reference policy, offering insights into how performance varies with the number of samples. We also study the regret gap, i.e., the gap between the expected reward of the optimal (tilted) policy and the SBoN policy. Our theoretical and empirical findings show that smoothing helps SBoN mitigate reward overoptimization, especially when the quality of the proxy reward is low.

Submission Number: 81