SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

Srivatsa R Kundurthy, Clara Na, Michael Handley, Zach Kirshner, Chen Bo Calvin Zhang, Manasi Sharma, Emma Strubell, John Ling

Keywords: spreadsheet, arena, evaluation, task formulation, structured artifact, structured data, Excel, preference evaluation, finance, domain

TL;DR: We introduce SpreadsheetArena, a platform for evaluating end-to-end spreadsheet generation capabilities of LLMs, with implications across domains such as professional finance

Abstract:

Large language models (LLMs) are increasingly tasked with producing and manipulating structured artifacts.
We consider the task of end-to-end **spreadsheet generation**, where language models are prompted to produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language.
We introduce **SpreadsheetArena**, a platform for evaluating models' performance on the task via blind pairwise preference votes of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary substantially across use cases and prompts, often in ways that are difficult to formalize.
Compared to general chat or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex considerations around interactivity and layout. Among other findings, we observe that stylistic, structural, and functional features of preferred spreadsheets vary substantially across use cases, and expert evaluations of spreadsheets for finance prompts suggests that even highly ranked arena models do not reliably produce spreadsheets aligned with domain-specific best practices.
Our hope is that our work prompts further study of end-to-end spreadsheet generation as a challenging and interesting class of complex, open-ended tasks for LLMs. Our live arena is hosted at https://spreadsheetarena.ai.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 92