Improving Composition of Sentence Embeddings through the Lens of Statistical Relational Learning

Damien Sileo, Tim Van de Cruys, Camille Pradel, Philippe Muller

Abstract: Various NLP problems -- such as the prediction of sentence similarity, entailment, and discourse relations -- are all instances of the same general task: the modeling of semantic relations between a pair of textual elements. We call them textual relational problems. A popular model for textual relational problems is to embed sentences into fixed size vectors and use composition functions (e.g. difference or concatenation) of those vectors as features for the prediction. Meanwhile, composition of embeddings has been a main focus within the field of Statistical Relational Learning (SRL) whose goal is to predict relations between entities (typically from knowledge base triples). In this work, we show that textual relational models implicitly use compositions from baseline SRL models. We show that such compositions are not expressive enough for several tasks (e.g. natural language inference). We build on recent SRL models to address textual relational problems, showing that they are more expressive, and can alleviate issues from simpler compositions. The resulting models significantly improve the state of the art in both transferable sentence representation learning and relation prediction.

Keywords: Statistical Relational Learning, Sentence Embedding, Composition functions, Natural Language Inference, InferSent, SentEval, ComplEx

TL;DR: We apply ideas from Statistical Relational Learning to compose sentence embeddings with more expressivity