What Would an LLM Do? Evaluating Policymaking Capabilities of Large Language Models

Pierre Le Coz, Jiaan Liu, Debarun Bhattacharjya, Georgina Curto, Serge Stinckwich

Keywords: Policy and Social Development; Low and Middle-Income Countries / Underserved Communities

TL;DR: the article evaluates LLMs to inform policymaking (through scenarios on homelessness aleviation in South Africa, Spain and the US) and links selected policies to agent-based model simulations to explore impact in urban settings.

Abstract: Large language models (LLMs) are increasingly being adopted in high-stakes domains. Their capacity to process vast amounts of unstructured data, explore flexible scenarios, and handle a diversity of contextual factors can make them uniquely suited to provide new insights for the complexity of social policymaking. This article evaluates whether LLMs’ are aligned with domain experts (and among themselves) to inform social policymaking on the subject of homelessness alleviation – a challenge affecting over 150 million people world- wide. We develop a novel benchmark comprised of decision scenarios with policy choices across four geographies (South Bend, USA; Barcelona, Spain; Johannesburg, South Africa; Macau SAR, China). The policies in scope are grounded in the conceptual framework of the Capability Approach for human development. We also present an automated pipeline that connects the benchmarked policies to an agent-based model, and we explore the social impact of the recommended policies through simulated social scenarios. The paper results reveal promising potential to leverage LLMs for social policy making. If responsible guardrails and contextual calibrations are introduced in collaboration with local domain experts, LLMs can provide humans with valuable insights, in the form of alternative policies at scale.

Submission Number: 16