TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns

Soma Onishi, Kenta Oono, Kohei Hayashi

Keywords: pre-training, tabular data, transfer learning, Transformer, masked autoencoder

TL;DR: TabRet is a pre-trainable Transformer that excels at adapting to new columns in tabular downstream tasks.

Abstract: We present TabRet, a pre-trainable Transformer-based model for tabular data. TabRet is designed to work on a downstream task that contains columns not seen in pre-training. Unlike other methods, TabRet has an extra learning step before fine-tuning called retokenizing, which calibrates feature embeddings based on the masked autoencoding loss. In experiments, we pre-trained TabRet with a large collection of public health surveys and fine-tuned it on classification tasks in healthcare, and TabRet achieved the best AUC performance on four datasets. In addition, an ablation study shows retokenizing and random shuffle augmentation of columns during pre-training contributed to performance gains. The code is available at https://github.com/pfnet-research/tabret.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/tabret-pre-training-transformer-based-tabular/code)