Semantic Correlation Adaptation for Union-Set Multi-label Image Recognition

Xinyu Wang, Tao Pu, Dongyu Zhang, Liang Lin

Abstract: Existing multi-label classification works are confined to fixed target categories, requiring lots of effort in collecting complete labels. However, annotating all relevant labels for novel categories is impracticable. To cope with this challenge, we investigate a new task, union-set multi-label image recognition (US-MLR), which allows a varying label space for each image rather than a fixed one (see Fig. 1). Beyond complementing missing labels, it further requires aligning semantic correlations among different splits. In this work, we propose a novel semantic correlation adaptation (SCA) framework, which firstly explores semantic correlations within each domain and across different domains to complement missing labels and then performs semantic correlation co-adaptation to alleviate the correlation inconsistency due to the domain gap. Comprehensive experiments on a new US-MLR benchmark and multiple MLR benchmarks demonstrate the effectiveness of the proposed SCA framework.