Query-based Image Captioning from Multi-context 360° Images

Koki Maeda, Shuhei Kurita, Taiki Miyanishi, Naoaki Okazaki

Submission Type: Regular Long Paper

Submission Track: Language Grounding to Vision, Robotics and Beyond

Keywords: Image Captioning, 360-degree image, Vision and Language

Abstract: A 360-degree image captures the entire scene without the limitations of a camera's field of view, which makes it difficult to describe all the contexts in a single caption. We propose a novel task called Query-based Image Captioning (QuIC) for 360-degree images, where a query (words or short phrases) specifies the context to describe. This task is more challenging than the conventional image captioning task, which describes salient objects in images, as it requires fine-grained scene understanding to select the contents consistent with user's intent based on the query. We construct a dataset for the new task that comprises 3,940 360-degree images and 18,459 pairs of queries and captions annotated manually. Experiments demonstrate that fine-tuning image captioning models further on our dataset can generate more diverse and controllable captions from multiple contexts of 360-degree images.

Submission Number: 4979