Conditional Diffusion Model for Open-ended Video Question Answering

Xinyue Liu, Jiahui Wan, Linlin Zong, Bo Xu

Abstract: Open-ended VideoQA presents a significant challenge due to the absence of fixed options, requiring the identification of the correct answer from a vast pool of candidate answers. Previous approaches typically utilize classifier or similarity comparison on fusion feature to yield prediction directly, lacking coarse-to-fine filtering on numerous candidates. Gradual refining the probability distribution of candidates can achieve more precise prediction. Thus, we propose the DiffAns model, which integrates the diffusion model to handle open-ended VideoQA task, simulating the gradual process by which humans answer open-ended question. Specifically, we first diffuse the true answer label into a random distribution (forward process). And under the guidance of answer-aware condition generated from video and question, the model iteratively denoises to obtain the correct probability distribution(backward process). This equips the model with the capability to progressively refine the random probability distribution of candidates, ultimately predicting the correct answer. We conduct experiments on three challenging open-ended VideoQA datasets, surpassing existing SoTA methods. Extensive experiments further explore and analyse the impact of each modules, as well as the design of diffusion model, demonstrating the effectiveness of DiffAns. Our code will be available.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Content] Multimodal Fusion

Relevance To Conference:

This paper presents the first attempt to introduce diffusion model to tackle Video Question Answering problem.
Video Question Answering (VideoQA) seek to explore techniques that combines video when answering question, representing one of the challenging multimodal tasks. It is more complex to interact with text and fuse for video than image, because of the temporal nature. Due to the importance in multimodal processing, VideoQA becomes a popular research field.
Open-ended VideoQA presents a significant challenge due to the absence of fixed options, requiring the identification of the correct answer from a vast pool of candidate answers. Previous approaches typically utilize classifier or similarity comparison on fusion feature to yield prediction directly, lacking coarse-to-fine filtering on numerous candidates.
We promoted the research in the field of Open-ended VideoQA by gradually refining answers through innovative solutions(e.i., diffusion model). We firmly believe that this work provides fundamental insights into the multimodal domain.

Submission Number: 3997