Sequential Emergence of Phonemic, Syntactic, and Semantic Representations in Neural Networks

Pierre Orhan, Pablo J. Diego Simon, Yves Boubenec, Emmanuel Chemla, Yair Lakretz, Jean-Remi King

Keywords: structure, linguistic, representations, decoding, self-supervised, speech

TL;DR: We propose that self-supervised models instantiate phoneme, semantic and syntactic structure, and measure the emergence of these 3 linguistic codes.

Abstract:

During language acquisition, children successively learn to categorize phonemes, identify words, and combine them with syntax to form new meaning.
While the development of this behavior is well characterized, we still lack a unifying computational framework to explain its underlying neural representations.
Here, we investigate whether and when phonemic, lexical and syntactic representations emerge in the activations of artificial neural networks during their training.
Our results show that both speech- and text-based models follow a sequence of learning stages: during training, their neural activations successively build subspaces, where the geometry of the neural activations represents phonemic, lexical, and syntactic structure.
While this developmental trajectory qualitatively relates to children's, it is quantitatively different: These algorithms indeed require 2 to 4 orders of magnitude more data for these neural representations to emerge.
Together, these results show conditions under which major stages of language acquisition spontaneously emerge, and hence delineate a promising path to understand the computations underpinning language acquisition.

Primary Area: interpretability and explainable AI

Submission Number: 11775