Block Redundancy in Tabular Foundation Models for Clinical Applications

Irene Cannistraci, Julia E Vogt

Keywords: Tabular Foundation Models, In-Context Learning, Clinical Machine Learning, Efficient Inference, Transformer Compression; Linear Approximation, Electronic Health Records, Resource-Constrained Deployment, Clinical Decision Support

TL;DR: We simplify Tabular Foundation Models by replacing most of their blocks with a simple linear translator, enabling efficiency without sacrificing performance

Abstract: In-context-learning Tabular Foundation Models (TFMs) are uniquely suited to clinical use because they require no training on patient data, however their architectures still demand GPU inference and discourage the on-premise deployment that hospitals need. Simpler alternatives (e.g. Gradient-Boosted Decision Tree) run on CPU and often match their performance, but at the cost of manual feature engineering, preprocessing, and hyperparameter tuning that transformer models avoid by design. In this paper we show that it is possible to simplify TFMs, by substituting up to ∼94% of the blocks using a closed-form linear translator, while preserving downstream performance and requiring minimal compute. We show results on eight clinical benchmarks of different sizes (including MIMIC-III and multi-center eICU-CRD) and three TabZilla controls, for classification (binary and multi-class) and regression.

Submission Number: 165