Transfer Learning with Deep Tabular ModelsDownload PDF

26 Sept 2022, 09:27 (modified: 15 Nov 2022, 03:26)TRL @ NeurIPS 2022 OralReaders: Everyone
Keywords: tabular data, transfer learning, tabular models, representation learning
TL;DR: We find that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods when downstream data is limited.
Abstract: Recent work on deep learning for tabular data demonstrates the strong performance of deep tabular models, often bridging the gap between gradient boosted decision trees and neural networks. Accuracy aside, a major advantage of neural models is that they are easily fine-tuned in new domains and learn reusable features. This property is often exploited in computer vision and natural language applications, where transfer learning is indispensable when task-specific training data is scarce. In this work, we explore the benefits that representation learning provides for knowledge transfer in the tabular domain. We conduct experiments in a realistic medical diagnosis test bed with limited amounts of downstream data and find that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods. We further compare the supervised and self-supervised pretraining strategies and provide practical advice on transfer learning with tabular models. Finally, we propose a pseudo-feature method for cases where the upstream and downstream feature sets differ, a tabular-specific problem widespread in real-world applications.
Slides: pdf
0 Replies

Loading