Discovering Latent Network Topology in Contextualized Representations with Randomized Dynamic Programming

Yao Fu, Mirella Lapata

Keywords: latent structures, dynamic programming, approximate inference, randomization, memory efficiency, contextualized representations, network topology, paraphrase generation, bertology

Abstract: The discovery of large-scale discrete latent structures is crucial for understanding the fundamental generative processes of language. In this work, we use structured latent variables to study the representation space of contextualized embeddings and gain insight into the hidden topology of pretrained language models. However, existing methods are severely limited by issues of scalability and efficiency as working with large combinatorial spaces requires expensive memory consumption. We address this challenge by proposing a Randomized Dynamic Programming (RDP) algorithm for the approximate inference of structured models with DP-style exact computation (e.g., Forward-Backward). Our technique samples a subset of DP paths reducing memory complexity to as small as one percent. We use RDP to analyze the representation space of pretrained language models, discovering a large-scale latent network in a fully unsupervised way. The induced latent states not only serve as anchors marking the topology of the space (neighbors and connectivity), but also reveal linguistic properties related to syntax, morphology, and semantics. We also show that traversing this latent network yields unsupervised paraphrase generation.

One-sentence Summary: We scale up DP-based inference with randomization then use it to discover a latent network within BERT representation space.