Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset

Anonymous

Abstract:

The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game commentary is not enough to fully understand the game situation. It will be richer by including diverse multimodal esports information, including audiences’ talks/emotions, game audio, and game match event information.
This paper introduces GAME-MUG, a new multimodal game situation understanding and audience-engaged commentary generation dataset, and its strong baseline. Our dataset is collected from 2020-2022 LOL game live streams from YouTube and Twitch, and includes multimodal esports game information, including text, audio, and time-series event logs, for detecting the game situation. In addition, we also propose a new audience conversation augmented commentary dataset by covering the game situation and audience conversation understanding and introducing a robust joint multimodal dual learning model as a baseline. We examine the model's game situation/event understanding ability and commentary generation capability to show the effectiveness of the multimodal aspects coverage and the joint integration learning approach.

Paper Type: long

Research Area: NLP Applications

Languages Studied: English,Korean,Chinese

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.