CataTQA: A Benchmark for Tool-Augmented LLM Question Answering over Heterogeneous Catalysis Tables

Qiang Cui, Ludi Wang, Yang Gao, Xueqing Chen, Wenjuan Cui, Yi Du

Keywords: Scientific Question Answering​​, Benchmark​, Tool Invocation​, LLMs

TL;DR: CataTQA, a novel benchmark with auxiliary tools, addresses LLMs' limitations in scientific QA by evaluating their ability to retrieve and interpret complex experimental data.

Abstract: Despite their success in general question answering, large language models (LLMs) struggle with hallucinations and inaccurate reasoning in scientific domains. A major challenge stems from experimental data, which are often stored in external sources like supplementary materials and domain-specific databases. These tables are large, heterogeneous, and semantically complex, making them difficult for LLMs to interpret. While external tools show promise, current benchmarks fail to assess LLMs' ability to navigate this data-particularly in locating relevant tables, retrieving key columns, interpreting experimental conditions, and invoking tools.To address this gap, we introduce CataTQA, a new benchmark for **Cata**lytic materials **T**able **Q**uestion and **A**nswering. CataTQA features an automated dataset framework and four auxiliary tools. We evaluate tool-enhanced LLMs across five dimensions: table location, column retrieval, condition analysis, tool calling, and question answering, identifying their strengths and weaknesses.Our work sets a new benchmark for evaluating LLMs in scientific fields and paves the way for future advancements. All data and code are publicly available on GitHub.

Primary Area: datasets and benchmarks

Submission Number: 16077