Transfer Learning with Deep Tabular ModelsDownload PDF

Published: 21 Oct 2022, Last Modified: 08 Sept 2024TRL @ NeurIPS 2022 OralReaders: Everyone
Keywords: tabular data, transfer learning, tabular models, representation learning
TL;DR: We find that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods when downstream data is limited.
Abstract: Recent work on deep learning for tabular data demonstrates the strong performance of deep tabular models, often bridging the gap between gradient boosted decision trees and neural networks. Accuracy aside, a major advantage of neural models is that they are easily fine-tuned in new domains and learn reusable features. This property is often exploited in computer vision and natural language applications, where transfer learning is indispensable when task-specific training data is scarce. In this work, we explore the benefits that representation learning provides for knowledge transfer in the tabular domain. We conduct experiments in a realistic medical diagnosis test bed with limited amounts of downstream data and find that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods. We further compare the supervised and self-supervised pretraining strategies and provide practical advice on transfer learning with tabular models. Finally, we propose a pseudo-feature method for cases where the upstream and downstream feature sets differ, a tabular-specific problem widespread in real-world applications.
Slides: pdf
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/transfer-learning-with-deep-tabular-models/code)
0 Replies

Loading