SAMoE-VAE: A Tabular Foundation Model with a Schema-Aware Mixture-of-Experts Variational Autoencoder

Xiaofeng Lin, Chunhe Wang, MINRUI GUI, Namjoon Suh, Jessup Byun, Oscar Chen, Zachary DeSoto, Yehyeon Cho, Guang Cheng

Keywords: tabular data, mixture-of-experts, transfer learning, foundation model, representation learning

Abstract:

Foundation models have revolutionized vision and language, yet tabular learning still depends on bespoke, per-dataset pipelines. A key challenge in developing a uniform representation that enables foundation model is \emph{schema mismatch}: real-world tables contain diverse column types: numeric, categorical, text, datetime, whose semantics vary across datasets. We frame cross-tabular representation learning as a weakly supervised, multi-modal problem, leveraging the readily available schema metadata that accompanies each table.
We propose SAMoE-VAE, a schema-aware Mixture-of-Experts VAE that:
(i) assigns separate experts to numeric, categorical, text, and datetime columns;
(ii) fuses expert posteriors via a schema-conditioned Product-of-Experts(MoPoE);
(iii) produces a probabilistic latent embedding space that drives accurate downstream prediction and schema-aware generation.
To train at scale, we curate \textbf{Meta-T4}, a 1.2-million-table corpus augmented with LLM-generated text metadata. Extensive experiments show that SAMoE-VAE outperforms prior art in tabular foundation models on representation learning benchmarks, yielding higher downstream accuracy and improved sample efficiency.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 23521