BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

Evan R Antoniuk, Shehtab Zaman, Tal Ben-Nun, Peggy Li, James Diffenderfer, Busra Sahin, Obadiah Hersh Smolenski, Everett Grethel, Tim Hsu, Anna Hiszpanski, Kenneth Chiu, Bhavya Kailkhura, Brian Van Essen

Keywords: Deep Learning, AI4Science, Benchmark, Molecular property, Out-of-distribution generalization

TL;DR: BOOM is a standardized benchmark for evaluating the out-of-distribution (OOD) performance of chemical machine learning models.

Abstract: Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel molecules. Discovering novel molecules requires accurate out-of-distribution (OOD) predictions, but ML models struggle to generalize OOD. Currently, no systematic benchmarks exist for molecular OOD prediction tasks. We present BOOM, $\textbf{b}$enchmarks for $\textbf{o}$ut-$\textbf{o}f$-$\textbf{d}$istribution $\textbf{m}$olecular property predictions: a chemically-informed benchmark for OOD performance on common molecular property prediction tasks. We evaluate over 150 model-task combinations to benchmark deep learning models on OOD performance. Overall, we find that no existing model achieves strong generalization across all tasks: even the top-performing model exhibited an average OOD error 3$\times$ higher than in-distribution. Current chemical foundation models do not show strong OOD extrapolation, while models with high inductive bias can perform well on OOD tasks with simple, specific properties. We perform extensive ablation experiments, highlighting how data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation impact OOD performance. Developing models with strong OOD generalization is a new frontier challenge in chemical ML. This open-source benchmark is available at https://github.com/FLASK-LLNL/BOOM

Dataset URL: https://github.com/FLASK-LLNL/LLNL-10k-Dataset

Code URL: https://github.com/FLASK-LLNL/BOOM

Primary Area: AL/ML Datasets & Benchmarks for physics (e.g. climate, health, life sciences, physics, social sciences)

Submission Number: 1090