DeLLMphi: A Multi-Turn Method for Multi-Agent Forecasting

Andrew Robert Williams, Martin Weiss, Victoria Feere, Nasim Rahaman, Hugo Larochelle

Keywords: multi-agent forecasting, expert elicitation, probabilistic forecasting, large language models, multi-turn interaction, consensus formation, event forecasting, in-context learning, judgment aggregation

TL;DR: DeLLMphi simulates the Delphi method with LLMs as experts and mediator. Example-based elicitation improves forecasts, while multi-turn interaction and mediator feedback surface disagreements and enhance accuracy.

Abstract: The Delphi method is a structured forecasting process that engages experts in iterative prediction and reflection. Each round, experts submit forecasts to a mediator, receive an aggregated and synthesized response highlighting key arguments, and update their forecasts based on collective insight. However, Delphi panels are labour intensive, slow and hard to reproduce, requiring diverse knowledgeable participants to engage periodically across weeks or months. To address these constraints, we propose **DeLLMphi**, a forecasting method that replaces human experts and mediators with LLMs. We show (i) that providing example superforecaster reasoning traces and predictions helps to elicit more accurate forecasts from LLM experts, (ii) that the mediator plays the crucial role of surfacing different lines of reasoning and points of disagreement, and (iii) that multiple rounds and experts lead to better forecasts, showing that multi-turn interaction is key to DeLLMphi.

Submission Number: 204