Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes

Haoyu Wang, Hongming Zhang, Yueguan Wang, Yuqian Deng, Muhao Chen, Dan Roth

Submission Type: Regular Short Paper

Submission Track: Commonsense Reasoning

Keywords: Event Granularities, Essentiality, Event Processes, Goals, Steps

TL;DR: This paper studies the essentiality of step events towards goal events, using a high-quality corpus from WikiHow, and finds that current models significantly lag behind human understanding, highlighting the need for further research.

Abstract: Natural language often describes events in different granularities, such that more coarse-grained (goal) events can often be decomposed into fine-grained sequences of (step) events. A critical but overlooked challenge in understanding an event process lies in the fact that the step events are not equally important to the central goal. In this paper, we seek to fill this gap by studying how well current models can understand the essentiality of different step events towards a goal event. As discussed by cognitive studies, such an ability enables the machine to mimic human’s commonsense reasoning about preconditions and necessary efforts of daily-life tasks. Our work contributes with a high-quality corpus of (goal, step) pairs from a community guideline website WikiHow, where the steps are manually annotated with their essentiality w.r.t. the goal. The high IAA indicates that humans have a consistent understanding of the events. Despite evaluating various statistical and massive pre-trained NLU models, we observe that existing SOTA models all perform drastically behind humans, indicating the need for future investigation of this crucial yet challenging task.

Submission Number: 4462