Transferring Domain-Agnostic Knowledge in Video Question Answering

Tianran Wu, Noa Garcia, Mayu Otani, Chenhui Chu, Yuta Nakashima, Haruo Takemura

Abstract:

Video question answering (VideoQA) is designed to answer a given question based
on a relevant video clip. The current available large-scale datasets have made it possible
to formulate VideoQA as the joint understanding of visual and language information.
However, this training procedure is costly and still less competent with human performance. In this paper, we investigate a transfer learning method by the introduction of
domain-agnostic knowledge and domain-specific knowledge. First, we develop a novel
transfer learning framework, which finetunes the pre-trained model by applying domainagnostic knowledge as the medium. Second, we construct a new VideoQA dataset with
21,412 human-generated question-answer samples for comparable transfer of knowledge. Our experiments show that: (i) domain-agnostic knowledge is transferable and (ii)
our proposed transfer learning framework can boost VideoQA performance effectively.