Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks
Keywords: tabular, representation learning, deep learning, gradient boosted decision trees, benchmarking
TL;DR: We conduct an initial investigation of sketching and feature-selection for prior-data fitted networks such as TabPFN, noting differences between them and conventional tabular models.
Abstract: Tabular classification has traditionally relied on supervised algorithms, which estimate the parameters of a prediction model using its training data. Recently, Prior-Data Fitted Networks such as TabPFN have successfully learned to classify tabular data in-context: the model parameters are designed to classify new samples based on labelled training samples given after the model training. While such models show great promise, their applicability to real-world data remains limited due to the computational scale needed. We conduct an initial investigation of sketching and feature-selection methods for TabPFN, and note certain key differences between it and conventionally fitted tabular models.
Submission Number: 37