Pareto-Optimal Fronts for Benchmarking Symbolic Regression Algorithms

Kei Sen Fong, Mehul Motani

Abstract: Symbolic Regression (SR) algorithms select expressions based on prediction performance while also keeping the expression lengths short to produce explainable white box models. In this context, SR algorithms can be evaluated by measuring the extent to which the expressions discovered are Pareto-optimal, in the sense of having the best R-squared score for a given expression length. This evaluation is most commonly done based on relative performance, in the sense that an SR algorithm is judged on whether it Pareto-dominates other SR algorithms selected in the analysis, without any indication on efficiency or attainable limits. In this paper, we explore absolute Pareto-optimal (APO) solutions instead, which have the optimal tradeoff between the multiple SR objectives, for 34 datasets in the widely-used SR benchmark, SRBench, by performing exhaustive search. Additionally, we include comparisons between eight numerical optimization methods. We extract, for every dataset, an APO front of expressions that can serve as a universal baseline for SR algorithms that informs researchers of the best attainable performance for selected sizes. The APO fronts provided serves as an important benchmark and performance limit for SR algorithms and is made publicly available at: https://github.com/kentridgeai/SRParetoFronts

Lay Summary: Symbolic regression is a field of machine learning that learns concise mathematical expressions for prediction. Most symbolic regression studies compare algorithms against each other, but it can be hard to know how close any of them get to the best possible formulas. In this work, we exhaustively search over simple mathematical expressions (up to a fixed size) for over 30 commonly used symbolic regression datasets to identify the absolute ‘best-accuracy-for-a-given-length’ trade-off, which we call the Pareto optimal front. This means that, for each allowed formula size, we aim to find an expression that maximizes predictive accuracy, rather than just outperforming other existing methods. We also examine how different numerical optimizers (e.g., BFGS, Powell, etc.) affect these results and show that, in practice, they produce very similar outcomes. By sharing these fronts, we hope to offer a helpful reference: researchers can see how close their own methods come to these best attainable targets (with some caveats) for short, interpretable equations. In this way, our work provides a clear baseline for benchmarking and suggests where there is still room to explore and improve symbolic regression approaches.

Link To Code: https://github.com/kentridgeai/SRParetoFronts

Primary Area: General Machine Learning->Supervised Learning

Keywords: Symbolic Regression, Benchmark

Submission Number: 14182