$\text{CIS}^2$: A Simplified Commonsense Inference Evaluation for Story Prose

Bryan Li, Lara J. Martin, Chris Callison-Burch

Keywords: story generation, commonsense reasoning, language models, evaluation methods

TL;DR: We introduce $\text{CIS}^2$, a simplified method for training and evaluating contextual commonsense inference that abstracts away language generation.

Abstract:

Contextual Commonsense Inference (CCI) is the problem of inferring causal relations between the events of a text, such as a story. Like other commonsense reasoning tasks, CCI is a problem of language understanding, rather than language generation. We show that prior work, in using language generation to perform CCI, trains models that struggle on the CCI task in isolation. This conflation of tasks is further exacerbated by evaluating with word-matching based metrics such as BLEU. In order to isolate CCI from language generation, we reframe CCI as a classification problem. Our system, which we call $\text{CIS}^2$, forces the model to focus on CCI directly by providing it the original text of the story to use for understanding while having it generate only the bare minimum: indices to sentences.
We look at the GLUCOSE (Mostafazadeh et al. 2020) dataset and compare against their task for predicting CCI between story sentences. We find that models trained on $\text{CIS}^2$index labels achieve a 4.3% higher CCI accuracy than those trained for generating full phrases, such as in the original GLUCOSE task.

Published: No

Archival: No