GAMformer: Exploring In-Context Learning for Generalized Additive Models

Andreas C Mueller, Julien Siems, Harsha Nori, Rich Caruana, Frank Hutter

Keywords: generalized additive models, gam, in-context learning, icl, interpretable machine learning, glassbox machine learning

TL;DR: We demonstrate how to estimate Generalized Additive Models using in-context learning.

Abstract:

Generalized Additive Models (GAMs) are widely recognized for their ability to create fully interpretable machine learning models for tabular data. Traditionally, training GAMs involves iterative learning algorithms, such as splines, boosted trees, or neural networks, which refine the additive components through repeated error reduction.
In this paper, we introduce *GAMformer*, the first method to leverage in-context learning to estimate shape functions of a GAM in a single forward pass, representing a significant departure from the conventional iterative approaches to GAM fitting.
Building on previous research applying in-context learning to tabular data, we exclusively use complex, synthetic data to train GAMformer, yet find it to extrapolate well to real-world data. Our experiments show that GAMformer performs on par with other leading GAMs across various classification benchmarks while generating highly interpretable shape functions.
Source code is available under this [link](https://anonymous.4open.science/r/900265cb0d5e51efb612/README.md.).

Track: Main track

Submitted Paper: Yes

Published Paper: No

Submission Number: 37