【Proposal】Transformer Models for Predicting Material Properties

Kaiwei Zhang, Juncheng Yu

Keywords: Transformer, Attention mechanism, Graph Neural Network

TL;DR: we propose to train a transformer model to predict material properties, while solving contribution assignment problem and capturing global features of crystals.

Abstract: Graph neural networks (GNNs) have been widely used to predict material properties. However, current GNN models cannot assign the contribution of each atom. Meanwhile, GNNs are poor at capturing global information, resulting in unsatisfactory predictions of certain properties. The widely used transformer may be suitable for crystals and could be used to tackle both problems. We propose to train a transformer for predicting crystal properties, followed by analyzing learned attentions to assign contributions and demonstrating the model’s global view. Further discussions could be made if successful. In terms of related works, we introduce research about proposed models, usage of attention mechanisms, common model variants as well as network applications.

Submission Number: 58