Transformer as a Neural Knowledge Graph

Yuki Nishihori, Yusei Ito, Yuta Suzuki, Ryo Igarashi, Yoshitaka Ushiku, Kanta Ono

Submission Track: Multi-Modal Data for Materials Design - Tiny Paper

Submission Category: AI-Guided Design

Keywords: Crystal structure, Contrastive learning, Multimodal learning, Knowledge graph

TL;DR: Propose a transformer-based neural knowledge graph to integrate contextual knowledge into contrastive learning between crystal structures and linguistic physical properties.

Abstract: In this study, we propose an effective contrastive learning method that bridges crystal structures with their linguistic properties (e.g., superconductor). Contrastive learning enables both the retrieval of crystal structures based on linguistic characteristics and the inference of linguistic properties from crystal structures, which are essential for accelerating materials discovery. However, a major challenge lies in the limitation of available datasets, which currently include only crystal structures paired with their corresponding article titles and abstracts. Because many papers depend on referenced works and shared domain knowledge—often explored in detail within the main text—titles and abstracts alone do not sufficiently capture the full characteristics of a crystal. To address this issue, we introduce a neural knowledge graph by incorporating a transformer into the text encoder of the existing contrastive learning framework, rather than expanding the dataset. This modification enables the model to dynamically incorporate related knowledge, thereby enhancing its representation of linguistic properties and facilitating more accurate correlations between crystal structures and their properties.

Submission Number: 52