ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue

Haoqin Tu, Yitong Li, Fei Mi, Zhongliang Yang

Submission Type: Regular Long Paper

Submission Track: Dialogue and Interactive Systems

Submission Track 2: Resources and Evaluation

Keywords: visual dialogue, multimodal dataset, knowledge-enhanced dialogue, pre-trained language model

TL;DR: Two currently the most fine-grained visual dialogue datasets with entity&turn-level images, and a unified multimodal dialogue system with either shared or separate encoder-decoder setup.

Abstract: Incorporating visual knowledge into text-only dialogue systems has become a potential direction to imitate the way humans think, imagine, and communicate. However, existing multimodal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To address these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (ReSee-$\texttt{WoW}$, ReSee-$\texttt{DD}$). We propose to explicitly split the visual knowledge into finer granularity ("turn-level" and "entity-level"). To further boost the accuracy and diversity of augmented visual information, we retrieve them from the Internet or a large image dataset. To demonstrate the superiority and universality of the provided visual knowledge, we propose a simple but effective framework ReSee to add visual representation into vanilla dialogue models by modality concatenations. We also conduct extensive experiments and ablations w.r.t. different model configurations and visual knowledge settings. Empirical, encouraging results not only demonstrate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model ReSee outperforms several state-of-the-art methods on automatic and human evaluations. By leveraging text and vision knowledge, ReSee can produce informative responses with real-world visual concepts. Our code is available at https://github.com/ImKeTT/ReSee.

Submission Number: 5864