Towards Decomposed Linguistic Representation with Holographic Reduced Representation

Jiaming Luo, Yuan Cao, Yonghui Wu

Abstract: The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures. In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced Representation (HRR). This allows us to inject structures directly into our word-level and chunk-level representations. Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics.

TL;DR: Holographic Reduced Representation enables language model to discover linguistic roles.