TripleThreat: Benchmarking Functional Sensitivity in Protein Representations with Paralog-Ortholog Triplets

Mohini K. Misra, Gowri Nayar, Emma Lundberg, Russ B Altman

Confirmation: I have read and agree with the workshop's policy on behalf of myself and my co-authors.

Track: tiny / short paper (2-4 pages excluding references; extended abstract format)

Keywords: benchmark dataset, protein representation learning, protein function, biological representation evaluation, paralogs/orthologs, protein language models, embedding evaluation

TL;DR: We created a new benchmark to test if protein representation spaces learn functional nuances by seeing how well they separate functionally divergent paralogs from functionally conserved orthologs, and we evaluated this on 5 protein language models.

Abstract: Understanding why certain sequence changes lead to functional change while others conserve function remains a central challenge in protein biology. A meaningful protein representation should be sensitive to these distinctions, so we introduce TripleThreat, a benchmark dataset to evaluate this capability. We construct test cases using natural examples of divergence and conservation in paralogs and orthologs, respectively, assembling protein–paralog–ortholog triplets. By controlling for confounders (sequence identity, length, and species) at varying levels, we generate six dataset subsets that trade off dataset size and stringency. Because protein language models (pLM) are popularly used as protein representations, we evaluate five widely used pLMs on this benchmark. We find that performance declines as confounding variables are more tightly controlled. Further, by varying confounder stringency, we identify which confounding signals overshadow functional signals; for example, an alignment-based pLM is observed to encode species identity more prominently than function. In sum, this work offers a framework to test whether protein representation spaces capture fine-grained functional relationships beyond confounding signals. We make our benchmark publicly available at https://github.com/mohinimisra26/triple-threat.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 86