Investigating Language Models for Supporting Complex Group Decisions

Sirui Zeng, Kanav Arora, Ming Yin, Xifeng Yan, Francesco Bullo, Ambuj Singh

Keywords: Group decision making, Large Language Model, Fairness, Reinforcement Learning, Multi Agent

TL;DR: We use LLMs, MCTS, and constraint solvers to coordinate group decisions, finding that reasoning models, numeric feedback, and solver support yield higher-quality, fairer solutions.

Abstract: Reaching consensus is a central challenge in group decision making as agreement needs to be balanced with diversity of perspectives. Recent AI advances have opened new possibilities for synthesizing complex information and facilitating consensus. We study group decision processes by modeling the complexity of the decision surface, defined by a set of decision problems, each with multiple options. Each solution yields a gain for every participant, and the objective of deliberation is to ensure fairness by equalizing participants’ profits. We explore multiple settings: whether gains are private, arbitrary numbers, or ordered sequences; whether the exact gain for each option is public; and whether group communication is expressed in natural language or numerically. Group coordination is facilitated by an AI agent powered by a large language model (LLM). We find that reasoning LLM models perform better than non-reasoning models and that a constraint solver (CPLEX) or a reinforcement learning agent (MCTS) improves the quality of the decision. The performance of reasoning models carries over when the participants rank order their preferences instead of assigning numeric scores. Numeric feedback leads to higher quality solutions than verbal feedback and is also better than when participants state their preference between two decisions. Our findings suggest that while LLMs show promise in facilitating consensus, there remains significant room for improvement in their ability to fully capture and reason over group consensus involving numerical outcomes.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 23848