Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks

Benjamin Feuer, Niv Cohen, Chinmay Hegde

Keywords: tabular, representation learning, deep learning, gradient boosted decision trees, benchmarking

TL;DR: We conduct an initial investigation of sketching and feature-selection for prior-data fitted networks such as TabPFN, noting differences between them and conventional tabular models.

Abstract: Tabular classification has traditionally relied on supervised algorithms, which estimate the parameters of a prediction model using its training data. Recently, Prior-Data Fitted Networks such as TabPFN have successfully learned to classify tabular data in-context: the model parameters are designed to classify new samples based on labelled training samples given after the model training. While such models show great promise, their applicability to real-world data remains limited due to the computational scale needed. We conduct an initial investigation of sketching and feature-selection methods for TabPFN, and note certain key differences between it and conventionally fitted tabular models.

Submission Number: 37