A Simpler and More Generalizable Story Detector using Verb and Character Features

Mark Finlayson

Abstract:

Story detection is the task of determining
whether or not a unit of text contains a
story. Prior approaches achieved a max-
imum performance of 0.66 F1, and did
not generalize well across different cor-
pora. We present a new state-of-the-art
detector that achieves a maximum per-
formance of 0.75 F1 (a 14% improve-
ment), with significantly greater general-
izability than previous work. In partic-
ular, our detector achieves performance
above 0.70 F1 across a variety of combi-
nations of lexically different corpora for
training and testing, as well as dramatic
improvements (up to 4,000%) in perfor-
mance when trained on a small, disfluent
data set. The new detector uses two basic
types of features–ones related to events,
and ones related to characters–totaling 283
specific features overall; previous detec-
tors used tens of thousands of features,
and so this detector represents a significant
simplification along with increased perfor-
mance.