Exploring the linear separablity of syntactic and semantic information in BERT embeddings

Anonymous

Abstract: Relations between syntax and semantics are not readily agreed upon. We seek to explore how representations of syntax and semantic information sets manifest in BERT embeddings, particularly the degree of the linear separability of each other in BERT embeddings by applying Iterative Nullspace Projection (INLP) to decompose BERT embeddings into syntactic and semantic subspaces. We also investigate how important the linear component corresponding to one information set is to solving a classification task that targets the other information set. Our results show that both syntactic and semantics informations are not linearly represented in BERT embeddings. Therefore INLP fails separate syntactic and semantic space from BERT embeddings and does not provide interpretable results. The results also indicate a factor of consideration when applying INLP, regarding the rank of the projection matrix.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP