Anchor & Transform: Learning Sparse Representations of Discrete Objects

Paul Pu Liang, Manzil Zaheer, Yuan Wang, Amr Ahmed

TL;DR: We propose a general method to learn sparse representations of discrete objects that is scalable, flexible, end-to-end trainable, and allows the user to easily incorporate domain knowledge about object relationships.

Abstract: Learning continuous representations of discrete objects such as text, users, and items lies at the heart of many applications including text and user modeling. Unfortunately, traditional methods that embed all objects do not scale to large vocabulary sizes and embedding dimensions. In this paper, we propose a general method, Anchor & Transform (ANT) that learns sparse representations of discrete objects by jointly learning a small set of anchor embeddings and a sparse transformation from anchor objects to all objects. ANT is scalable, flexible, end-to-end trainable, and allows the user to easily incorporate domain knowledge about object relationships (e.g. WordNet, co-occurrence, item clusters). ANT also recovers several task-specific baselines under certain structural assumptions on the anchors and transformation matrices. On text classification and language modeling benchmarks, ANT demonstrates stronger performance with fewer parameters as compared to existing vocabulary selection and embedding compression baselines.

Keywords: sparse representation learning, discrete inputs, natural language processing

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2003.08197/code)