Adversarial Learning for Fine-Grained Image Search

Kevin Lin, Fan Yang, Qiaosong Wang, Robinson Piramuthu

Abstract: Fine-grained image search is still a challenging problem due to the difficulty in capturing subtle differences regardless of pose variations of objects from fine-grained categories. In practice, a dynamic inventory with new fine-grained categories adds another dimension to this challenge. In this work, we propose an end-to-end network, called FGGAN, that learns discriminative representations by implicitly learning a geometric transformation from multi-view images for fine-grained rigid object retrieval. We integrate a generative adversarial network (GAN) that can automatically handle complex view and pose variations by converting them to a canonical view without any predefined transformations. Moreover, in an open-set scenario, our network is able to better match rigid objects from unseen and unknown fine-grained categories. Extensive experiments on the public CompCars dataset and a newly collected dataset have demonstrated the effectiveness of the proposed method in both closed-set and open-set scenarios.