In-Context Learning of Soft Nearest Neighbor Classifiers for Intelligible Tabular Machine Learning

Mykhailo Koshil, Matthias Feurer, Katharina Eggensperger

Keywords: In-Context Learning, Soft Nearest Neighbor, Intelligibility, Kernel Regression, Kernel Learning, Representation Learning, PFN

Abstract: With in-context learning foundation models like TabPFN excelling on small supervised tabular learning tasks, it has been argued that "boosted trees are not the best default choice when working with data in tables''. However, such foundation models are inherently black-box models that do not provide interpretable predictions. We introduce a novel learning task to train ICL models to act as a nearest neighbor algorithm, which enables intelligible inference and does not decrease performance empirically.

Include In Proceedings: Yes

Submission Number: 21