Health-Oriented Multimodal Food Question Answering

Jianghai Wang, Menghao Hu, Yaguang Song, Xiaoshan Yang

Abstract: Health-oriented food analysis has become a research hotspot in recent years because it can help people keep away from unhealthy diets. Significant progress has been made in recipe retrieval, food recommendation, nutrition and calorie estimation. However, existing works still cannot well balance the individual preference and the health. Multimodal food question and answering (Q &A) has a great potential in practical applications, but it is still not well studied. In this paper, we build a health-oriented multimodal food Q &A dataset (MFQA) with 9K question and answer pairs based on a multimodal food knowledge graph collected from a food-sharing website. In addition, we propose a knowledge-based multimodal food Q &A framework, which consists of three important parts: encoder module, retrieval module, and answer module. Extensive experimental results on the MFQA dataset demonstrate the effectiveness of our method. The code and dataset are available at https://github.com/Wjianghai/HMFQA .