Unified sampling framework and benchmarking of sequence- and structure-based protein models

Aviv Spinner, Pascal Notin, Samuel P Berry, Dana Cortade, Zach Sisson, Svetlana P Ikonomova, David Ross, Debora Susan Marks

Keywords: protein design, generative models, sampling, benchmarking, experimental validation

TL;DR: We benchmark diverse generative protein models under a unified sampling framework and collect experimental data on over 13K sequences.

Abstract: Generative models are increasingly used for protein design, but the lack of standardized evaluation frameworks limits comparison across model classes and hinders translation to experimental success. Here, we introduce a unified sampling and benchmarking framework that enables controlled sequence generation across alignment, protein language, and structure-based models, and apply it to Tobacco etch virus (TEV) protease. Across hundreds of thousands of designed sequences, different models explore distinct regions of sequence space with no clear computational selection metrics to assess enzymatic function. Experimental evaluation reveals large differences in functional outcomes, ranging from non-functional variants to sequences with ~9-fold higher activity than wildtype. Machine learning-designed libraries achieve a 39.32\% hit rate (percentage of variants matching or exceeding wildtype activity) compared to 6.06\% for an error-prone PCR baseline. Structure-based models perform best overall, with hit rates of 74.4\% and 66.8\% for ESM-IF1 and ProteinMPNN, respectively. Commonly used selection metrics do not strongly correlate with experimental activity, highlighting a gap between in silico evaluation and enzyme function. Together, these results establish a generalizable framework for benchmarking generative protein models and demonstrate the necessity of experimental validation for guiding model development and sequence prioritization.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 57