Interpreting Language Models Through Knowledge Graph Extraction

Vinitra Swamy, Angelika Romanou, Martin Jaggi

Keywords: explainable AI, NLP, knowledge graphs, language models, BERT, Transformers

TL;DR: We extract knowledge graphs from sequential stages of language model training to diagnose models' strengths and weaknesses.

Abstract: Transformer-based language models trained on large text corpora have enjoyed immense popularity in the natural language processing community and are commonly used as a starting point for downstream tasks. While these models are undeniably useful, it is a challenge to quantify their performance beyond traditional accuracy metrics. In this paper, we compare BERT-based language models through snapshots of acquired knowledge at sequential stages of the training process. Structured relationships from training corpora may be uncovered through querying a masked language model with probing tasks. We present a methodology to unveil a knowledge acquisition timeline by generating knowledge graph extracts from cloze "fill-in-the-blank" statements at various stages of RoBERTa's early training. We extend this analysis to a comparison of pretrained variations of BERT models (DistilBERT, BERT-base, RoBERTa). This work proposes a quantitative framework to compare language models through knowledge graph extraction (GED, Graph2Vec) and showcases a part-of-speech analysis (POSOR) to identify the linguistic strengths of each model variant. Using these metrics, machine learning practitioners can compare models, diagnose their models' behavioral strengths and weaknesses, and identify new targeted datasets to improve model performance.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2111.08546/code)