Attribute Driven Zero-Shot Classification and Segmentation

Shu Yang, Yemin Shi, Yaowei Wang, Jing Wang, Zesong Fei

Abstract: Zero-shot classification and segmentation aims to recognize and segment objects of unseen classes. The attribute information, such as color, shape, part and material, is usually used for zero-shot classification. Moreover, we observe that this kind of attribute information could also be helpful in the segmentation task. On this basis, we propose an Attribute-Segmentation-Attribute (ASA) framework to address the zero-shot classification and segmentation problem. In the framework, a multi-task model is pre-trained to capture category and attribute features simultaneously. Then, a two-branch fully convolutional structure is built on the pre-trained model and fine-tuned for segmentation task. Finally, the extracted class-unseen object is recognized with the segmentation-assisted attribute prediction and a class-attribute matrix. Experimental results on the public bench-mark datasets indicate that the proposed ASA framework out-performs the state-of-the-art methods for both classification and segmentation tasks.