GAMformer: Exploring In-Context Learning for Generalized Additive Models

Andreas C Mueller, Julien Siems, Harsha Nori, David Salinas, Arber Zela, Rich Caruana, Frank Hutter

Keywords: tabular data, generalized additive models, GAM, in-context learning, synthetic data, transformer, interpretability

TL;DR: GAMformer uses in-context learning to estimate Generalized Additive Models in a single forward pass, performing comparably to traditional iterative approaches like EBMs and spline based GAMs while maintaining interpretability.

Abstract: Generalized Additive Models (GAMs) are widely recognized for their ability to create fully interpretable machine learning models for tabular data. Traditionally, training GAMs involves iterative learning algorithms, such as splines, boosted trees, or neural networks, which refine the additive components through repeated error reduction. In this paper, we introduce GAMformer, the first method to leverage in-context learning to estimate shape functions of a GAM in a single forward pass, representing a significant departure from the conventional iterative approaches to GAM fitting. Building on previous research applying in-context learning to tabular data, we exclusively use complex, synthetic data to train GAMformer, yet find it to extrapolate well to real-world data. Our experiments show that GAMformer performs on par with other leading GAMs across various classification benchmarks while generating highly interpretable shape functions.

Submission Number: 68