Deep Tabular Learning via Distillation and Language Guidance

Ruohan Wang, Wenhao Fu, Carlo Ciliberto

Abstract: Tabular data is arguably one of the most ubiquitous data structures in application domains such as science, healthcare, finance and manufacturing. Given the recent success of deep learning (DL), there has been a surge of new DL models for tabular learning. However, despite the efforts, tabular DL models still clearly trail behind tree-based approaches. In this work, we propose DisTab, a novel framework for tabular learning based on the transformer architecture. Our method leverages model distillation to mimic the favorable inductive biases of tree-based models, and incorporates language guidance for more expressive feature embeddings. Empirically, DisTab outperforms existing tabular DL models and is highly competitive against tree-based models across diverse datasets, effectively closing the gap with these methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Incorporated all changes requested by the AE for the minor revision.

Code: https://github.com/RuohanW/DisTab

Assigned Action Editor: ~Kenta_Oono1

Submission Number: 3221