How Should LLMs Assist Humans in Table Unionability Tasks?

Nina Klimenkova, Roee Shraga, Erin Solovey

Keywords: Table Unionability, Data Discovery, LLM-assisted decision-making, Human-in-the-loop, Agentic data systems

TL;DR: How an LLM's assistance is presented shapes whether it helps or harms human table-unionability decisions and how well people's confidence tracks correctness.

Abstract: Large language models (LLMs) increasingly take part in data discovery decisions such as table unionability—judging whether two tables can be meaningfully combined. Once framed as a model prediction or human-labeling problem, unionability is increasingly settled through LLM-assisted decision-making, where a person decides but a model's suggestion becomes one more input. The design question is then not only whether to provide assistance, but how to present it. We report a pilot survey in which participants judge a table pair, then may revise after seeing assistance in one of three forms: a bare recommendation, a recommendation with an explanation, or one that also conveys the model's expressed uncertainty. Across 90 decisions, aggregate accuracy barely moved, yet a transition-level view shows assistance corrected some errors while introducing a comparable number of new ones, and assistance form was associated with differences in both answer revision and confidence calibration. These preliminary findings suggest treating assistance form as a policy variable in future agentic data-discovery systems that expose, suppress, or qualify a suggestion according to the predicted risk of harmful reliance.

Submission Number: 8