Rethinking Table Instruction Tuning

Naihao Deng, Rada Mihalcea

Keywords: table instruction tuning, table understanding

TL;DR: We reveal the limitations of the current table instruction tuning, explore the technical details, and fine-tune our own model based on our findings.

Abstract:

Recent advances in table understanding have focused on instruction-tuning large language models (LLMs) for table-related tasks.
However, existing research has overlooked the impact of hyperparameter choices and lacks a comprehensive evaluation of the out-of-domain table understanding ability and the general capabilities of these table LLMs.
In this paper, we evaluate these abilities in existing table LLMs, and reveal significant declines in both out-of-domain table understanding and general capabilities compared to their base models.
Through systematic analysis, we show that hyperparameters, such as learning rate, can significantly influence both table-specific and general capabilities.
Contrary to the existing table instruction-tuning works, we demonstrate that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities.
Based on our findings, we introduce **TAMA**, a **TA**ble LLM instruction-tuned from LLa**MA** 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks, while maintaining strong out-of-domain generalization and general capabilities.
Our findings highlight the potential for reduced data annotation costs and more efficient model development through careful hyperparameter selection.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8813