$\textbf{\emph{CLMSM}}$: A Multi-Task Learning Framework for Pre-training on Procedural Text

Abhilash Nandy, Manav Nitin Kapadnis, Pawan Goyal, Niloy Ganguly

Submission Type: Regular Long Paper

Submission Track: NLP Applications

Submission Track 2: Commonsense Reasoning

Keywords: pre-training, procedural reasoning, contrastive learning, masked language modeling, multi-task learning, nlp

TL;DR: We propose CLMSM, a pre-training framework that uses multi-task learning with contrastive learning and mask-step modeling objectives for downstream procedural NLP tasks, and outperforms baselines on in-domain as well as open-domain tasks.

Abstract: In this paper, we propose ***CLMSM***, a domain-specific, continual pre-training framework, that learns from a large set of procedural recipes. ***CLMSM*** uses a Multi-Task Learning Framework to optimize two objectives - a) Contrastive Learning using hard triplets to learn fine-grained differences across entities in the procedures, and b) a novel Mask-Step Modelling objective to learn step-wise context of a procedure. We test the performance of ***CLMSM*** on the downstream tasks of tracking entities and aligning actions between two procedures on three datasets, one of which is an open-domain dataset not conforming with the pre-training dataset. We show that ***CLMSM*** not only outperforms baselines on recipes (in-domain) but is also able to generalize to open-domain procedural NLP tasks.

Submission Number: 2159