RoViST: Learning Robust Metrics for Visual Storytelling

Anonymous

Paper Link: https://openreview.net/forum?id=bwIOahh3kHO

Paper Type: Long paper (up to eight pages of content + unlimited references and appendices)

Abstract: Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on $n$-gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.

Presentation Mode: This paper will be presented virtually

Virtual Presentation Timezone: UTC+10

Copyright Consent Signature (type Name Or NA If Not Transferrable): Eileen Wang

Copyright Consent Name And Address: The University of Sydney, NSW, Australia