Transformers Also Struggle with Extrapolating Ill-Behaved Learning Curves

Adelina-Andreea Cazacu, Cheng Yan, Sayak Mukherjee, Tom Julian Viering

Track: Type A (Regular Papers)

Keywords: Machine Learning, Learning Curve, LCDB, Extrapolation, Prior-Fitted Networks, Transformer, In-Context Learning

Abstract: Sample-wise learning curves capture how model performance scales with data, helping practitioners predict performance at larger data scales and allocate resources more effectively. While both parametric models and neural networks are used for learning curve extrapolation, their comparative advantages remain underexplored. In this work, we present a systematic comparison of Real Data Learning Curve Prior-Fitted Networks (Real Data LC-PFN) against three parametric models (POW4, MMF4, and WBL4) using the Learning Curves Database 1.1 (LCDB 1.1). Our analysis reveals that Real Data LC-PFN consistently achieves stronger extrapolation accuracy across diverse generalization scenarios, with notable advantages when only limited observations are available. However, while it handles the commonly observed, well-behaved monotone and convex curve shapes well, performance on ill-behaved learning curves, such as dipping, remains less competitive than parametric models. Our findings highlight the importance of context-aware model selection rather than universal approaches.

Serve As Reviewer: ~Tom_Julian_Viering1

Submission Number: 67