TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

Rasa Hosseinzadeh, Alex Labach, Zexin Xue, Shuyi Han, Valentin Thomas, Anthony L. Caterini

Keywords: Tabular Foundation Models, In-Context Learning, Long Contexts, Tabular Modelling, Tabular Data

TL;DR: We speed up TabDPT while maintaining the same prediction quality.

Abstract:

Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity.
However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial.
We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval.
By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster.

Submission Number: 125