GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning

Zhi Chen, Yadan Luo, Sen Wang, Jingjing Li, Zi Huang

Abstract: Generalized Zero-Shot Learning (GZSL) aims to recognize images not only for seen classes but also for unseen ones by transferring semantic-visual relationships from the seen to the unseen classes. It is an intuitive solution to take the advantage of generative models to hallucinate realistic unseen samples based on the knowledge learned from the seen classes. However, due to the generation shifts, the synthesized samples by most existing methods may drift from the real distribution of the unseen data. To address this issue, we propose a novel flow-based generative framework that consists of multiple conditional affine coupling layers for learning unseen data generation. Specifically, we investigate and address three essential problems that trigger the generation shifts, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semantic inconsistency</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">variance collapse</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">structure disorder</i> . First, to improve the reflection of the semantic information in the generated samples, we proactively embed the semantic information into the transformation in each conditional affine coupling layer. Second, to promote the intrinsic feature variance of the unseen classes, we introduce a boundary sample mining strategy with entropy maximization to discover ambiguous visual variants of semantic prototypes and hereby calibrate the decision boundary of the classifiers. Third, a relative positioning strategy is proposed to revise the attribute embeddings, guiding which to fully preserve the inter-class geometric structure and further avoid structure disorder in the semantic space. Extensive experimental results on four GZSL benchmark datasets demonstrate that GSMFlow achieves the state-of-the-art performance on GZSL.