DEMON24: ACM MM24 Demonstrative Instruction Following Challenge

Zhiqi Ge, Juncheng Li, Qifan Yu, Wei Zhou, Siliang Tang, Yueting Zhuang

Abstract: We introduce the DEMON Challenge, defined as a benchmark for demonstrative instruction following, to ACM Multimedia 2024. The DEMON Challenge aims to assess the ability of models and systems to comprehend demonstrative instructions consisting of multiple, interleaved, and multimodal context that demonstrate the required information to complete a task. These instructions are curated from a diverse range of multi-modal datasets, spanning various fields and scenarios, to ensure comprehensive coverage and challenge diversity. The challenge details and participation information are available on the https://dcdmllm.github.io/DEMON-challenge/.