Injecting Prior Knowledge into Image Caption GenerationDownload PDF

20 Jul 2020 (modified: 30 Jul 2020)ECCV 2020 Workshop VIPriors Blind SubmissionReaders: Everyone
  • Keywords: Image Captioning, Prior, Attention, Regularization
  • Supplementary Material: zip
  • TL;DR: In this paper, we propose to improve the performance of the state-of-the-art image captioning models by incorporating two sources of prior knowledge: Conditional Latent Topic Attention and a Sentence Prior.
  • Abstract: Automatically generating natural language descriptions from an image is a challenging problem in artificial intelligence that requires a good understanding of the visual and textual signals and the correlations between them. The state-of-the-art methods in image captioning struggle to approach human level performance, especially when data is limited. In this paper, we propose to improve the performance of the state-of-the-art image captioning models by incorporating two sources of prior knowledge: (i) a conditional latent topic attention, that uses a set of latent topics as an anchor to generate highly probable words and, (ii) a regularization technique that exploits the inductive biases in syntactic and semantic structure of captions and improves the generalization of image captioning models. Our experiments validate that our method produces more human interpretable captions and also leads to significant improvements on the MSCOCO dataset in both the full and low data regimes.
3 Replies

Loading