MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning

Arundhati Banerjee, Soham Rajesh Phade, Stefano Ermon, Stephan Zheng

Abstract: We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. Moreover, the principal should be few-shot adaptable and minimize the number of interventions, because interventions are often costly. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents with different learning strategies and reward functions. We validate this approach step-by-step. First, in a Stackelberg setting with a best-response agent, we show that meta-learning enables quick convergence to the theoretically known Stackelberg equilibrium at test time, although noisy observations severely increase the sample complexity. We then show that our model-based meta-learning approach is cost-effective in intervening on bandit agents with unseen explore-exploit strategies. Finally, we outperform baselines that use either meta-learning or agent behavior modeling, in both $0$-shot and $1$-shot settings with partial agent information.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Sebastian_Tschiatschek1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1337