Robust Deep Neural Networks for Heterogeneous Tabular Data

Vadim Borisov, Klaus Broelemann, Enkelejda Kasneci, Gjergji Kasneci

Keywords: Deep Learning, Deep Neural Networks, Tabular Data, Gradient Boosted Decision Trees

Abstract: Although deep neural networks (DNNs) constitute the state-of-the-art in many tasks based on image, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF, a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, TreeDrivenEncoder, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that a DNN can use to improve the predictive performance. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6\% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.

One-sentence Summary: Making deep neural networks robust to tabular data quality issues by utilizing decision tree ensembles for data encoding.