User-adaptive image retrieval via fusing pointwise and pairwise labels

Lin Chen, Peng Zhang, Baoxin Li

Abstract: In many Web image retrieval applications, adapting the retrieval results according to some model of the user is a desired feature as the returned images can be made specifically relevant to a user’s needs. Making retrieval user-adaptive faces several practical challenges, including the ambiguity of user query, the lack of user-adaptive training data, and lack of proper mechanisms for supporting adaptive learning. To address some of these challenges, we propose a hybrid learning strategy that fuses knowledge from both pointwise and pairwise training data into one framework for attribute-based, user-adaptive image retrieval. An online learning algorithm is developed for updating the ranking performance based on user feedback. The framework is also derived into a kernel form allowing easy application of kernel techniques. We use both synthetic and real-world datasets to evaluate the performance of the proposed algorithm. Comparison with other state-of-the-art approaches suggests that our method achieves obvious performance gains over ranking and zero-shot learning. Further, our online learning algorithm was found to be able to deliver much better performance than batch learning, given the same elapsed running time.