Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning

Cheol Woo Kim, Jai Moondra, Shresth Verma, Madeleine Pollack, Lingkai Kong, Milind Tambe, Swati Gupta

Abstract: In many real-world applications of Reinforcement Learning (RL), deployed policies have varied impacts on different stakeholders, creating challenges in reaching consensus on how to effectively aggregate their preferences. Generalized $p$-means form a widely used class of social welfare functions for this purpose, with broad applications in fair resource allocation, AI alignment, and decision-making. This class includes well-known welfare functions such as Egalitarian, Nash, and Utilitarian welfare. However, selecting the appropriate social welfare function is challenging for decision-makers, as the structure and outcomes of optimal policies can be highly sensitive to the choice of $p$. To address this challenge, we study the concept of an $\alpha$-approximate portfolio in RL, a set of policies that are approximately optimal across the family of generalized $p$-means for all $p \in [-\infty, 1]$. We propose algorithms to compute such portfolios and provide theoretical guarantees on the trade-offs among approximation factor, portfolio size, and computational efficiency. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of our approach in summarizing the policy space induced by varying $p$ values, empowering decision-makers to navigate this landscape more effectively.

Lay Summary: In many real-world applications of reinforcement learning, a single policy can affect different stakeholders in very different ways. This creates a challenge: how do we fairly aggregate these diverse preferences when making policy deployment decisions? A common approach is to use a social welfare function, which combines the utilities of all stakeholders into a single measure of overall benefit. However, there is no universally "right" social welfare function. Each one reflects a different notion of fairness, and choosing among them can be difficult. To address this challenge, we develop an algorithm that constructs a small portfolio of policies that are approximately optimal across an entire family of social welfare functions known as the generalized $p$-means. This allows decision-makers to choose from a concise set of high-quality policies without committing to a single fairness principle in advance. Our method helps visualize how stakeholder outcomes shift as fairness preferences change, enabling more transparent and informed decision-making.

Link To Code: https://github.com/jaimoondra/approximation-portfolios-for-rl/

Primary Area: Reinforcement Learning->Planning

Keywords: Reinforcement learning, multi-objective optimization, p-means aggregation

Submission Number: 6737