Learning shared manifold representation of images and attributes for generalized zero-shot learning

Masahiro Suzuki, Yusuke Iwasawa, Yutaka Matsuo

Abstract: Many of the zero-shot learning methods have realized predicting labels of unseen images by learning the relations between images and pre-defined class-attributes. However, recent studies show that, under the more realistic generalized zero-shot learning (GZSL) scenarios, these approaches severely suffer from the issue of biased prediction, i.e., their classifier tends to predict all the examples from both seen and unseen classes as one of the seen classes. The cause of this problem is that they cannot properly learn a mapping to the representation space generalized to the unseen classes since the training set does not include any unseen class information. To solve this, we propose a concept to learn a mapping that embeds both images and attributes to the shared representation space that can be generalized even for unseen classes by interpolating from the information of seen classes, which we refer to shared manifold learning. Furthermore, we propose modality invariant variational autoencoders, which can perform shared manifold learning by training variational autoencoders with both images and attributes as inputs. The empirical validation of well-known datasets in GZSL shows that our method achieves the significantly superior performances to the existing relation-based studies.

Keywords: zero-shot learning, variational autoencoders

Data: [AwA](https://paperswithcode.com/dataset/awa-1), [AwA2](https://paperswithcode.com/dataset/awa2-1), [CUB-200-2011](https://paperswithcode.com/dataset/cub-200-2011), [aPY](https://paperswithcode.com/dataset/apy)