A Transformer Model for Symbolic Regression towards Scientific Discovery

Published: 28 Oct 2023, Last Modified: 17 Nov 2023NeurIPS2023-AI4Science OralEveryoneRevisionsBibTeX
Keywords: symbolic regression, transformer, scientific discovery
TL;DR: A new transformer model for Symbolic Regression towards Scientific Discovery: state-of-the-art performances and discussion of the current and future challenges.
Abstract: Symbolic Regression (SR) searches for mathematical expressions which best describe numerical datasets. This allows to circumvent interpretation issues inherent to artificial neural networks, but SR algorithms are often computationally expensive. This work proposes a new Transformer model aiming at Symbolic Regression particularly focused on its application for Scientific Discovery. We propose three encoder architectures with increasing flexibility but at the cost of column-permutation equivariance violation. Training results indicate that the most flexible architecture is required to prevent from overfitting. Once trained, we apply our best model to the SRSD datasets (Symbolic Regression for Scientific Discovery datasets) which yields state-of-the-art results using the normalized tree-based edit distance, at no extra computational cost.
Submission Track: Original Research
Submission Number: 9