Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence

Anonymous

Paper Link: https://openreview.net/forum?id=-JMVJNCRGPg

Paper Type: Long paper (up to eight pages of content + unlimited references and appendices)

Abstract: The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs (p is true iff ¬p is false), is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLMs’ LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, named meaning-matching, designed to directly learn a meaning text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach outperforms our previous counterparts despite its time and resource efficiency.

Copyright Consent Signature (type Name Or NA If Not Transferrable): Myeongjun Jang

Copyright Consent Name And Address: Department of Computer Science, University of Oxford (Wolfson Building, ParksRoad, Oxford, UK, OX1 3QG)