Discovering Universal Geometry in Embeddings with ICA

Hiroaki Yamagiwa, Momose Oyama, Hidetoshi Shimodaira

Submission Type: Regular Long Paper

Submission Track: Interpretability, Interactivity, and Analysis of Models for NLP

Submission Track 2: Multilinguality and Linguistic Diversity

Keywords: Embeddings, Independent Component Analysis, Principal Component Analysis, Cross-lingual, Interpretability, Isotropy, Whitening

TL;DR: We studied the independent semantic components that are universal to embeddings of various languages and image models.

Abstract: This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.

Submission Number: 898