Language Models Are Good Tabular Learners
Abstract: Transformer-based language models have become the de facto standard in natural language processing. However, they underperform in the tabular data domain compared to traditional tree-based methods. We posit that current models fail to achieve the full potential of language models due to (i) heterogeneity of tabular data; and (ii) challenges faced by the model in interpreting numerical values. Based on this hypothesis, we propose the Tabular Domain Transformer (TDTransformer) framework. TDTransformer has distinct embedding processes for different types of columns. The alignment layers for different column-types transform these embeddings to a common space. Besides, TDTransformer adapts piece-wise linear encoding for numerical values for better performance. We test the proposed method on 76 real-world tabular classification datasets from the OpenML benchmark. Extensive experiments indicate that TDTransformer significantly improves the state-of-the-art methods.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: - We added the comparison of the model size in Appendix Section **Comparison of Computational Cost**. Besides, we added a sentence in main manuscript
> The comparison of computational costs is reported in Appendix Section A.6.
- We added the implementation details in Appendix Section **Implemntation Details on Baseline Methods**.
> When obtaining the performance of baseline methods, we also use constant hyperparameters. Implementation details can be found in Appendix Section A.5.
- We replaced the anynomous github link with the released code link [TDTransformer Repository](https://github.com/Zhenhan-Huang/TDTransformer).
Code: https://github.com/Zhenhan-Huang/TDTransformer
Supplementary Material: pdf
Assigned Action Editor: ~Frederic_Sala1
Submission Number: 4120
Loading