Decomposing Co-occurrence Matrices into Interpretable Components as Formal Concepts

Anonymous

TL;DR: We uncovered that mathematically identified formal concepts in a word co-occurrence matrix coincide with interpretable categories, which elucidates the mechanism of word embeddings.

Abstract:

This study addresses the interpretability of word representations through an investigation of a co-occurrence matrix.
Employing the mathematical methodology of Formal Concept Analysis, we reveal an underlying structure that is amenable to human interpretation.
Furthermore, we unveil the emergence of hierarchical and geometrical structures within word vectors as consequences of word usage.
Our results reveal a significant correspondence between semantic and vector spaces.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English