ChemDFM-X: towards large multimodal model for chemistry

Zihan Zhao, Bo Chen, Jingpiao Li, Lu Chen, Liyang Wen, Pengyu Wang, Zichen Zhu, Danyang Zhang, Yansi Li, Zhongyang Dai, Xin Chen, Kai Yu

Abstract: We proposed ChemDFM-X, a large multimodal model for chemistry. ChemDFM-X is a generalist model that has the ability to understand five of the most commonly used modalities in the field of chemistry, including structural modalities, image modalities, and characterization modalities. The evaluation results show that ChemDFM-X possesses the capabilities to comprehend chemical data in all five non-text modalities. This assists ChemDFM-X in outperforming other generalist models in a series of common chemical tasks and demonstrates the practical value of ChemDFM-X in chemistry research. It also has the potential for dealing with inputs of multiple modalities simultaneously, which is powerful in reaction-related tasks and will be further studied in our future work.