Paragraph-based Transformer Pre-training for Multi-Sentence Inference

Anonymous

Paper Link: https://openreview.net/forum?id=F2dhh0WG_Gt

Paper Type: Short paper (up to four pages of content + unlimited references and appendices)

Abstract: Inference tasks such as answer sentence selection (AS2) or fact verification are typically solved by fine-tuning transformer-based models as individual sentence-pair classifiers. Recent studies show that these tasks benefit from modeling dependencies across multiple candidate sentences jointly. In this paper, we first show that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks. We then propose a new pre-training objective that models the paragraph-level semantics across multiple input sentences. Our evaluation on three AS2 and one fact verification datasets demonstrates the superiority of our pre-training technique over the traditional ones for transformers used as joint models for multi-candidate inference tasks, as well as when used as cross-encoders for sentence-pair formulations of these tasks.

Presentation Mode: This paper will be presented in person in Seattle

Copyright Consent Signature (type Name Or NA If Not Transferrable): Siddhant Garg

Copyright Consent Name And Address: Amazon Services Inc., 1240 Rosecrans Avenue, Manhattan Beach, CA-90266