ExcelFormer: Making Neural Network Excel in Small Tabular Data Prediction

Jintai Chen, Jiahuan Yan, Danny Chen, Jimeng Sun, Jian Wu

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: tabular data, deep neural network architecture, supervised learning.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A method for tabular data classification and regression tasks, in supervised learning manner.

Abstract: Data organized in tables are omnipresent in real-world applications. Despite their strong performance on large-scale datasets, deep neural networks (DNNs) perform inferior on small-scale tabular data, which hinders the wider adoption of DNNs across domains. In this paper, we propose a holistic framework comprising a novel neural network architecture called ExcelFormer and two data augmentation approaches, which achieves high-precision prediction for supervised classification and regression tasks, particularly on small-scale tabular datasets. The core component of ExcelFormer is a novel "semi-permeable attention" coupled with a special initialization, which explicitly diminishes the impacts of uninformative features, thereby improving data-efficiency. The methodology insight behind two tabular data augmentation approaches, Feat-Mix and Hid-Mix, is to increase the training samples in a way accommodating the inherent irregularities of data patterns. Comprehensive experiments on diverse small-scale tabular datasets show that, our ExcelFormer consistently and substantially outperforms previous works, with no noticeable dataset type preference. Remarkably, we find the superiority of ExcelFormer extends to large datasets as well.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4888