Better by default: Strong pre-tuned MLPs and boosted trees on tabular data

David Holzmüller, Leo Grinsztajn, Ingo Steinwart

Keywords: tabular data, benchmark, default parameters, neural networks, deep learning, multilayer perceptron, gradient-boosted decision trees

TL;DR: We propose better default parameters for boosted decision trees and improved neural networks on tabular data, evaluate them on separate large benchmarks, and show that they can achieve excellent results with moderate runtime.

Abstract: For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.

Primary Area: Machine learning for other sciences and fields

Submission Number: 21034