Leveraging Foundation Models in Healthcare: A Distillation Approach to Interpretable Clinical Prediction

Hans Farrell Soegeng, Tristan Guérand, Thomas Peyrin

Track: Tiny Paper Track (Page limit: 3-5 pages)

Keywords: Tabular foundation models, few-shot learning, model distillation, interpretability, explainable AI, rule-based models, medical AI, clinical data analysis,

TL;DR: We propose a distillation pipeline for globally interpretable classification on scarce clinical data by leveraging foundation models, showing that our pipeline improves performance across 8 datasets from 4 to 256 shots.

Abstract: In data-scarce settings, learning accurate and interpretable models for high-stakes medical tabular classification remains a fundamental challenge, as healthcare decisions must be transparent and trustworthy. We propose a novel pipeline for few-shot tabular classification that distills the predictive power of large foundation models into globally interpretable student models. Given a tabular dataset, we generate synthetic data using CTGAN to approximate the underlying distribution. We then finetune high-capacity teacher models (TabPFN, TabM) on a small number of labeled examples and use them to pseudolabel the synthetic data. Finally, we train student explainer models (i.e., XGBoost, decision trees, Generalized Linear Rule Model (GLRM), and TTnet) on this pseudolabeled synthetic dataset. These student models are globally and exactly interpretable, yielding logical decision rules (e.g., disjunctive normal form) that fully reproduce their predictions. Evaluated across 8 clinical tabular tasks, our distilled models generally outperform baselines trained directly on the few-shot data, with improved ROC AUC scores across few-shot settings. This work demonstrates that foundation models can be effectively leveraged as teachers to produce small, transparent, and high-performing classifiers. Our approach advances the goal of reliable and interpretable machine learning in real-world settings where labeled data is limited.

Submission Number: 8