Towards Identification of Latent Structures in Language Embeddings

Ryunosuke Abe, Takatomi Kubo, Kazushi Ikeda

Keywords: Contrastive learning, CEBRA, Identifiability, Representation learning

Abstract:

This work aims to identify interpretable low-dimensional structures inherent in high-dimensional language embeddings. Prior studies suggest that linear independent component analysis (ICA) can transform embeddings into spaces with semantically meaningful axes, and similar approaches may extend beyond language to other modalities such as vision. As a natural extension, we consider nonlinear ICA to capture latent structure in nonlinear internal representations; however, generic nonlinear ICA suffers from the long-standing problem of identifiability. To address this, we adopt CEBRA, a contrastive-learning framework that achieves identifiability up to linear transformations by leveraging auxiliary variables.

In preliminary experiments using emotion labels as auxiliary variables, CEBRA maps sentence embeddings into a low-dimensional, linearly separable space, consistent with the view that its InfoNCE loss behaves as a multiclass discriminative objective under discrete labels. Moreover, across random initializations, the learned embeddings exhibit strong alignment up to linear transforms, providing empirical evidence for identifiability in practice. We discuss open questions regarding the choice of auxiliary variables, the interpretation of linearly equivalent solutions, and the topology of the learned low-dimensional manifolds. As a longer-term goal, we plan brain-encoding studies using fMRI to test whether the discovered structures correspond to neural representations involved in affective language processing.

Submission Number: 161