MatFusion: A Multi-Modal Framework Bridging LLMs and Structural Embeddings for Experimental Materials Property Prediction

Yuwei Wan, Yuqi An, Dongzhan Zhou, Jiahao Dong, Chunyu Kit, Wenjie Zhang, Bram Hoex, Tong Xie, Yingheng Wang

Submission Track: Multi-Modal Data for Materials Design - Tiny Paper

Submission Category: AI-Guided Design

Keywords: Multi-modal learning, Structural embeddings, Materials science, Embedding

Abstract: The scarcity of experimental data in materials science often necessitates property predictions based on large-scale simulations, which may suffer from accuracy and reliability limitations. Uni-modal representations derived from simulated structures inherently incorporate approximations—such as the choice of exchange-correlation functional in Density Functional Theory (DFT)—which constrain machine learning models in capturing complex experimental characterizations. In this work, we propose a novel multi-modal framework, MatFusion} that integrates embeddings from domain-specific large language models (LLMs) and structural models to enhance the prediction of experimental material properties. Our approach combines LLM-derived embeddings of material compositions with graph-based structural representations, achieving a 9.15\% reduction in mean absolute error (MAE) for experimental bandgap prediction. By leveraging both experiential knowledge from materials science literature and first-principles structural information, our framework transcends traditional representation constraints, offering a powerful paradigm for improving experimental materials property predictions.

AI4Mat Journal Track: Yes

Submission Number: 63