Towards Scaling Laws for Language Model Powered Evolutionary Algorithms: Case Study on Molecular Optimization

Tigran Fahradyan, Filya Geikyan, Philipp Guevorguian, Hrant Khachatrian

Track: Machine learning: computational method and/or computational results

Nature Biotechnology: No

Keywords: scaling laws, test-time compute, evolutionary algorithms, molecular optimization

TL;DR: We developed a parametric law for modeling the scaling dynamics of language model-enhanced evolutionary algorithms.

Abstract:

The improvement of large language models (LLMs) came from scaling
pretraining. However, a new scaling paradigm emerged called test-time compute, which
uses more computation at the inference time of language models to get better results. There have been extensive works suggesting various test-time compute scaling strategies, but the modeling of the scaling dynamics of these methods is still an open research question. In this work we try to bridge this gap, developing a parametric law for language model-enhanced evolutionary algorithms depending on the language model parameters (N) and number of evolutionary iterations (k) used. We show that in molecular optimization tasks, our law is able to accurately extrapolate 2.5 times in N and k. Additionally, our law suggests that there is a tradeoff between N and k, which we validate by matching the performance of a 3.2B model with an 8.5 times smaller 380M model using 2.3 times more evolutionary algorithms steps.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Presenter: ~Tigran_Fahradyan1

Format: Maybe: the presenting author will attend in person, contingent on other factors that still need to be determined (e.g., visa, funding).

Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.

Submission Number: 88