Sequential learning for sketch-based 3D model retrieval

Hairui Yang, Yu Tian, Caifei Yang, Zhihui Wang, Lei Wang, Haojie Li

Abstract: Sketch-based 3D model retrieval suffers large visual discrepancy between 3D models and 2D sketches. Most existing methods directly project samples from both modalities into a same semantic embedding space to alleviate the discrepancy. We argue that simultaneous learning of those two modalities would restrict the discrimination of 3D model representation, resulting in inferior retrieval results. In this work, we propose a novel sequential learning (SL) framework for sketch-based 3D model retrieval to learn 3D model representation and 2D sketch representation separately and sequentially. Specifically, the SL framework is composed of two modules, 3D model network (3DMN) and 2D sketch network (2DSN). Firstly, we train 3DMN with a discriminative loss formulated only on 3D models to promote discrimination. Then, the learned representations of 3D models guide 2DSN to learn discriminative 2D sketch representations. In the second phase, we further mine the implicit fine-grained class information of 3D models by unsupervised clustering algorithms. An alignment loss is formulated on 2D sketches and corresponding fine-grained class centers of 3D models. Extensive experiments on three large-scale benchmark datasets for 3D model retrieval validate the efficacy of the proposed SL framework and fine-grained class representations.

External IDs:dblp:journals/mms/YangTYWWL22