Benchmarking Intelligent LLM Agents for Conversational Data Analysis

Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Bowen Qin, Xiaodong Li, Chenhao Ma, Jian-Guang Lou, Reynold Cheng

Keywords: Conversational Data Analysis, Large Language Models, Benchmark, Multi-agent Environment, Adaptive Interaction Reflection, Decision-making

TL;DR: We introduce Tapilot-Crossing, a new benchmark for evaluating Large Language Models on conversational data analysis, along with an adaptive reflection strategy (ACR) that improves model performance by up to 44.5%.models.

Abstract:

Conversational Tabular Data Analysis, a collaboration between humans and machines, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic conversational logs for tabular data analysis hinder comprehensive quantitative evaluation of Large Language Models (LLMs) in this task. To mitigate this issue, we introduce **Tapilot-Crossing**, a new benchmark to evaluate LLMs on conversational data analysis. **Tapilot-Crossing** contains 1024 conversations, covering 4 practical scenarios: *Normal*, *Action*, *Private*, and *Private Action*. Notably, **Tapilot-Crossing** is constructed by an economical multi-agent environment, **Decision Company**, with few human efforts. This environment ensures efficiency and scalability of generating new conversational data. Our comprehensive study, conducted by data analysis experts, demonstrates that Decision Company is capable of producing diverse and high-quality data, laying the groundwork for efficient data annotation. We evaluate popular and advanced LLMs in **Tapilot-Crossing**, which highlights the challenges of conversational tabular data analysis. Furthermore, we propose **A**daptive **C**onversation **R**eflection (**ACR**), a self-generated reflection strategy that guides LLMs to **learn from successful histories**.
Experiments demonstrate that **ACR** can evolve LLMs into effective conversational data analysis agents, achieving a relative performance improvement of up to 44.5%.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8746