ORION-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning

Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay kumar Sankarapu

Submission Type: Short paper (4 pages)

Keywords: Tabular Foundation Models, Tabular In-Context Learning, Tabular Data, AutoML

Abstract: Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL, have achieved state-of-the-art performance comparable to gradient-boosted trees (GBTs) without task-specific fine-tuning. However, current architectures exhibit key limitations: (1) single-scale feature processing that overlooks hierarchical dependencies, (2) dense attention with quadratic scaling in table width, and (3) strictly sequential processing that prevents iterative refinement and cross-component communication. To address these challenges, we introduce ORION-MSP, a tabular ICL architecture featuring three key innovations: multi-scale processing to capture hierarchical feature interactions, block-sparse attention combining windowed, global, and random patterns for scalable efficiency, and a Perceiver-style memory enabling safe bidirectional information flow across components. Across diverse benchmarks, ORION-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables, establishing a new standard for efficient tabular in-context learning.

Submission Number: 52