First-flexible interactive retrieval system for visual lifelog exploration

Nhat Hoang-Xuan, Hoang-Phuc Trang-Trung, Mai-Khiem Tran, Thanh-Cong Le, E-Ro Nguyen, Van-Tu Ninh, Tu-Khiem Le, Minh-Triet Tran

Abstract: Photos and videos are generated frequently in our daily activities. With a huge collection of visual data, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest for a wide variation of query types. This motivates us to develop and upgrade our solution for visual lifelog exploration. Our solution includes a web-based retrieval system and a series of best practices to assist users in using our system. This paper details the enhanced version of FIRST, our Flexible Interactive Retrieval SysTem for visual lifelog retrieval. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query, and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With a flexible system architecture, our system can easily integrate new modules to enhance its functionality. We conduct a user study to analyze the behaviors of novice and expert users of our system for different query scenarios. Finally, we propose best practice guidelines for users of our system to efficiently find events or moments of interest.