Know Yourself and Know Your Neighbour : A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetwork

Vasudevan Nedumpozhimana, John D. Kelleher

Abstract: Sentence representation learning is still an open challenge in Natural Language Processing. In this work, we propose a new self-supervised framework for learning sentence representations, using a special type of neural network called a recursive hypernetwork. Our proposed model composes the representation of a sentence from representations of words by applying a recursive composition through the parse tree. We maintain a separate syntactic and semantic representation, and the semantic composition is guided by the information from the syntactic representation. To train this model, we introduce a novel set of six self-supervised tasks. By analysing the performance on 7 probing tasks, we validate that the generated sentence representation encodes richer linguistic information than both averaging baselines and state-of-the-art alternatives. Furthermore, we assess the impact of the six proposed self-supervised training tasks through ablation studies. We also demonstrate that the representations generated by our model are stable for sentences of varying length and that the semantic composition operators adapt to different syntactic categories.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

The camera-ready version with the requested minor revision has the following modifications:

1. Tweaked the claim "We validate that the generated sentence representation encodes richer linguistic information than both averaging baselines and state-of-the-art alternatives" to align it with the empirical evidence.

2. Added the ablation study and modified the related claims.

3. Included the Llama results.

4. Included two recent models as baselines.

5. Incorporated other revisions agreed in the review response.

6. Deanonymized the paper.

7. Included the link to github page that contains the source code with a README file.

Code: https://github.com/vasudev2020/codi/tree/main

Assigned Action Editor: ~Eleni_Triantafillou1

Submission Number: 5251