Learning Procedural Dependencies from Self-Supervised Instruction Unshuffling

Anonymous

TL;DR: Develops a self-supervised training method for improving LM reasoning about instructional texts

Abstract: We develop a self-supervised method for improving the ability of language models to reason about the dependency structure of procedural texts. Previous work has explored using finetuned models to classify dependencies between procedure steps and construct flow-graphs using these dependencies. We improve upon these methods by introducing a self-supervised step-unshuffling training objective. By learning to map shuffled sequences of procedure steps to their correct order, our method improves the procedural reasoning abilities of language models. Through experiments we demonstrate that state-of-the-art models including GPT-4 perform poorly at the task of identifying step dependencies, and we also generate significant improvements using our step-unshuffling training objective, surpassing GPT-4 performance.

Paper Type: short

Research Area: Semantics: Sentence-level Semantics, Textual Inference and Other areas

Contribution Types: NLP engineering experiment

Languages Studied: English