LLaMat: Large Language Models for Materials Science

Vaibhav Mishra, Somaditya Singh, Mohd Zaki, Hargun Singh Grover, Santiago Miret, Mausam ., N M Anoop Krishnan

Submission Track: Short Paper

Submission Category: AI-Guided Design

Keywords: large language model, materials discovery, natural language processing, information extraction

TL;DR: Large Language Models for Materials Science using LLaMA-2 pretrained on 30B materials science token

Abstract: Large language models are versatile tools that have been recently used in the materials science domain for tasks ranging from information extraction to acting as scientific assistants in materials discovery. It is believed that using domain-specific large language models will help improve performance on such tasks. In this work, we address the challenge of efficiently accessing and utilizing vast textual knowledge in materials science using continued pre-training of Meta's LLaMA-2-7B on curated materials science texts, enhancing its domain-specific capabilities. We also developed LLaMat-Chat, an instruction fine-tuned variant of LLaMat that is tailored through a dataset of one million instruction-output pairs, enabling interactive applications and proficient performance in natural language processing tasks within materials science. We show that LLaMat achieves state-of-the-art performance on several information extraction tasks from materials science text. Since the pre-training corpus also included crystallographic information files, it will be interesting in future to evaluate the materials discovery applications of LLaMat.

Submission Number: 16