Bayesian Preference Elicitation with Language Models

Anonymous

Abstract: There is increasing interest in using language models (LMs) not only for answering queries but also for gathering information about the preferences of human users. This preference data can be used to fine-tune LMs via reward modeling or completing goal-oriented tasks. However, LMs have been shown to struggle with crucial aspects of preference learning: quantifying uncertainty, modeling mental states, and posing highly informative questions. These challenges have been addressed in other areas of machine learning, such as Bayesian Optimal Experimental Design (BOED), which focuses on designing informative queries within a well-defined feature space. But these methods, in turn, have historically been difficult to scale and apply to real-world problems (e.g. involving text and images), in which simply identifying the relevant features can be challenging. We introduce OPEN (Optimal Preference Elicitation with Natural language) a framework that uses BOED to guide the choice of informative questions and an LM to extract features and translate abstract BOED queries into natural language questions. By combining the flexibility of LMs with the precision of BOED, OPEN can optimize queries for informativity while remaining adaptable to real-world domains. Conducting user studies, OPEN outperforms existing LM- and BOED-based methods for preference elicitation.

Paper Type: long

Research Area: Dialogue and Interactive Systems

Contribution Types: NLP engineering experiment, Data analysis

Languages Studied: English