From Market to Dish: Multi-ingredient Image Recognition for Personalized Recipe Recommendation

Lin Zhang, Jianbo Zhao, Si Li, Boxin Shi, Ling-Yu Duan

Abstract: Recognition of food ingredients enables applications on recipe recommendation for developing a healthier eating habit. Existing ingredients recognition methods largely rely on ideal images captured in a controlled environment, while ingredients are usually displayed unorderly in a complex environment in the market. We propose the multi-ingredient recognition problem in the market and develop a Spatial Regularization Network (SRN) based method to solve it by using a newly collected multiple vegetable image dataset captured in the market. We further use the recognition result to develop a recipe recommendation system to satisfy the daily nutrition requirements and individual preference of each user. Experiments show that our multi-ingredient recognition outperforms previous methods over 14% in mAP and recommendation model shows an improvement of over 23% in HR@10.