Sparsely Connected Layers for Financial Tabular Data
Keywords: Tabular Data, Finance, Neural Networks, Deep Learning, XGBoost, Decision Trees
TL;DR: In this paper, we show that for the consumer financial tabular data, sparsely connected layers can increase neural network depth and reliably outperform gradient-boosted trees.
Abstract: While neural networks are the standard for unstructured data, such as images and text, their performance often lags behind more traditional machine learning models, like gradient-boosted trees, for tabular data. This is supported by academic studies, industry practices, and Kaggle competitions. In particular, there is no straightforward way to increase the number of layers in neural networks applied to tabular data — a key factor in their success with unstructured data. Deep fully connected networks suffer from the vanishing gradient problem, and convolutional layers and transformers are generally not directly applicable to tabular data. Special constructs, such as skip layers and attention mechanisms, have been adapted to tabular data models with limited success. In this paper, we show that for consumer financial tabular data, while standard two-layer neural networks typically underperform when compared to gradient-boosted trees, sparsely connected layers can increase network depth and reliably outperform gradient-boosted trees. The superior performance appears to stem from the sparse layers’ ability to reduce correlations in the input data, a common challenge in high-dimensional tabular data. Therefore, we are hopeful that this method could be applicable to other domains facing similar challenges.
Submission Number: 86
Loading