Tabular Data: Deep Learning is Not All You Need

Ravid Shwartz-Ziv, Amitai Armon

Keywords: AutoML, Tabular Data, Deep Learning, Tree Ensemble Models

TL;DR: We systematically compared XGBoost with deep-learning models for tabular data, considering accuracy, computational, and hyperparameter optimization time, and found that XGBoost outperformed them, but the ensemble of all models performed better.

Abstract: A key element of AutoML systems is setting the types of models that will be used for each type of task. For classification and regression problems with tabular data, the use of tree ensemble models (like XGBoost) is usually recommended. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use-cases. In this paper, we explore whether these deep models should be a recommended option for tabular data, by rigorously comparing the new deep models to XGBoost on a variety of datasets. In addition to systematically comparing their accuracy, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of the deep models and XGBoost performs better on these datasets than XGBoost alone.

Ethics Statement: Tabular data is the most common data type in real-world use-cases, so selecting the right modeling methods for it is of high importance, and our research may affect many applications. We also hope our findings will encourage more research on tabular data modeling, including both deep and classical methods, leading to a better understanding of this type of data and hopefully better results. Another aspect that our research highlights is the importance of rigorous comparisons between models, including consideration of hyperparameter optimization time and inference computational cost. We hope this will also impact future research in this field.