A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDINGDownload PDF

28 Nov 2022, 11:38 (modified: 21 Jul 2022, 19:52)ICLR 2017 PosterReaders: Everyone
TL;DR: a new model for extracting an interpretable sentence embedding by introducing self-attention and matrix representation.
Abstract: This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.
Keywords: Natural language processing, Deep learning, Supervised Learning
Conflicts: us.ibm.com, iro.umontreal.ca, umontreal.ca
15 Replies

Loading