Early Stopping Tabular In-Context Learning

Jaris Küken, Lennart Purucker, Frank Hutter

Keywords: Tabular Foundation Models, Inference Efficiency, Tabular Data

TL;DR: In-Context learning with tabular foundation models can be early stopped to improve inference efficiency.

Abstract:

Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetuning. However, their inference-time costs remain high, particularly for larger datasets.
To address this, we propose early-stopping the in-context learning process.
We achieve this by dynamically evaluating whether to stop in-context learning after each Transformer encoder layer.
Once stopped, we decode the embedding using a pre-trained layer-wise decoder.
Experiments across 34 small classification tasks size show that early stopping in-context learning accelerates inference by up to $\times1.3$ with negligible degradation in predictive performance. To assess scalability, we further evaluate our method on five larger classification tasks, achieving speedups of up to $\times2.2$. Our results demonstrate the potential of early exiting as an effective and practical strategy for improving the efficiency of tabular in-context learning.

Submission Number: 19