Can Test-time Computation Mitigate Reproduction Bias in Neural Symbolic Regression?

ICLR 2026 Conference Submission18283 Authors

Keywords: symbolic regression, transformer, test-time computation, self-verification, compositional generalization

TL;DR: We find that reproduction bias limits the search space for Transformer-based symbolic regression and explore test-time strategies to mitigate it.

Abstract: Symbolic regression aims to discover mathematical equations that fit given numerical data. It has been applied in various fields of scientific research, such as producing human-readable expressions that explain physical phenomena. Recently, Neural symbolic regression (NSR) methods that involve Transformers pre-trained on large-scale synthetic datasets have gained attention. While these methods offer advantages such as short inference time, they suffer from low performance, particularly when the number of input variables is large. In this study, we analyze the reasons for this limitation and suggest ways to improve NSR. We first provide a theoretical analysis showing that, under naive inference strategies, Transformers are unable to construct expressions in a compositional manner while verifying their numerical validity. Next, we explore how Transformers generate expressions in practice despite the lack of compositional generalizability. Our empirical analysis shows that the search space of NSR methods are greatly restricted due to reproduction bias, where the majority of generated expressions are merely copied from the training data. We finally examined if tailoring test-time strategies can reduce reproduction bias and improve numerical accuracy. We empirically demonstrate that providing additional information to the model at test time can significantly mitigate reproduction bias. On the other hand, we also found that reducing reproduction bias does not necessarily correlate with improved accuracy. These findings contribute to a deeper understanding of the limitation of NSR approaches and offer a foundation for designing more robust, generalizable symbolic regression methods.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 18283