TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data

Anonymous

Abstract: In this work, we address question answering (QA) over a hybrid of tabular and textual data, involving a variety of common content in reality like SEC filings, where discrete reasoning is often required. We consider harnessing the multi-step reasoning capabilities of large language models (LLMs) to tackle this problem, which have recently achieved remarkable success in many natural language tasks. To do this, we first abstract a Step-wise Pipeline for tabular and textual QA to help LLMs better execute multi-step inference, containing three key steps of Extractor, Reasoner and Executor. We initially design an instruction to validate the pipeline on GPT-4, demonstrating promising results. However, utilizing an online LLM like GPT-4 holds various challenges in terms of cost, latency, and data security risk, which motivates us to specialize smaller LLMs in this task. We then develop a TAT-LLM model by fine-tuning LLaMA 2 with the training data generated automatically from existing datasets following the Step-wise Pipeline. The experimental results have verified that our TAT-LLM model can outperform all compared models, including prior best fine-tuned models and very large-scale LLMs like GPT-4 on FinQA, TAT-QA and TAT-DQA benchmarks.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English