From Epoch to Sample Size: Developing New Data-driven Priors for Learning Curve Prior-Fitted Networks

Tom Julian Viering, Steven Adriaensen, Herilalaina Rakotoarison, Frank Hutter

Keywords: prior fitted networks, learning curves, training set size, epochs, training set size, prior, Bayesian

TL;DR: We develop 2 new datadriven LC-PFN priors for extrapolating learning curves to extrapolate learning curves that visualize performance versus training set size using curves from the LCDB

Abstract:

Learning Curve Prior-Fitted Networks (LC-PFNs) perform Bayesian learning curve extrapolation for epoch learning curves. This paper explores designing new priors for LC-PFNs, focusing on sample-size learning curves that relate training set size to performance. We use the Learning Curve Database (LCDB), which contains diverse learning curve data for machine learning models on tabular data, to develop two data-driven priors. The first method fits MMF4 and WBL4 parametric curve models to the LCDB and uses a Gaussian mixture model to represent the prior over parametric curve parameters. The second method directly trains the LCPFNs on the LCDB curves, which we call the Real Data LC-PFN. We set up a proper meta-learning curve extrapolation benchmark with cross-validation on the LCDB for a careful evaluation. We show that both proposed priors improve upon the original LC-PFN, with the Real Data LC-PFN providing best results, improving in 78% of the experiments upon the old prior for extrapolating learning curves. Our study illustrates how to systematically design new priors for LC-PFN's in a metalearning framework, opening up their use for various curve modeling tasks in machine learning and beyond.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

Submission Number: 25