Table Foundation Models: on knowledge pre-training for tabular learning

Myung Jun Kim, Félix Lefebvre, Gaëtan Brison, Alexandre Perez-Lebel, Gaël Varoquaux

Abstract: Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, *e.g.*, column name. The traditional approach combines column-specific data preparation with tree-based models that adapt to column specificities. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

First revision:
- Additional baseline of RealMLP and CARTE-B-TabPFNv2 in the experiments.
- Additional ablation on architecture, pre-training, and preprocessing components of TARTE with comparisons to CARTE pre-training.
- Clarifications on critical difference diagrams, datasets for domain specialization, and downstream variants of TARTE.
- Analysis on experiment results.
- Corrections on grammatical errors.

Final revision:
- Added credits for the logos of YAGO project and Wikidata.
- Added the link to the repository for implementation of the model.
- Deleted the supplementary materials containing the source codes, which can be found at the repository.
- General editing of all the figures and tables.

Code: https://github.com/soda-inria/tarte-ai

Assigned Action Editor: ~Kenta_Oono1

Submission Number: 4841