Location-Aware Visual Question Generation with Lightweight Models

Nicholas Collin Suwono, Justin Chen, Tun Min Hung, Ting-Hao Kenneth Huang, I-Bin Liao, Yung-Hui Li, Lun-Wei Ku, Shao-Hua Sun

Submission Type: Regular Long Paper

Submission Track: NLP Applications

Submission Track 2: Natural Language Generation

Keywords: Location-aware Visual Question Generation, Visual Question Generation, Question Generation, Lightweight Models

TL;DR: We propose a novel Location-aware Visual Question Generation (LocaVQG) task, create a dataset, and present a method that can learn a lightweight model to address the task.

Abstract: This work introduces a novel task, location-aware visual question generation (LocaVQG), which aims to generate engaging questions from data relevant to a particular geographical location. Specifically, we represent such location-aware information with surrounding images and a GPS coordinate. To tackle this task, we present a dataset generation pipeline that leverages GPT-4 to produce diverse and sophisticated questions. Then, we aim to learn a lightweight model that can address the LocaVQG task and fit on an edge device, such as a mobile phone. To this end, we propose a method which can reliably generate engaging questions from location-aware information. Our proposed method outperforms baselines regarding human evaluation (e.g., engagement, grounding, coherence) and automatic evaluation metrics (e.g., BERTScore, ROUGE-2). Moreover, we conduct extensive ablation studies to justify our proposed techniques for both generating the dataset and solving the task.

Submission Number: 2933