Beyond Direct Relationships: Exploring Multi-Order Label Pair Dependencies for Knowledge Distillation

Jingchao Wang, Zhengnan Deng, Tongxu Lin, Wenyuan Li, Shaobin Ling, Junyu Lin

Abstract: Multi-label image classification is crucial for a wide range of multimedia applications. To address the resource limitation issue, various knowledge distillation (KD) methods have been developed to transfer knowledge from a large network (referred to as the "teacher") to a small network (referred to as the "student"). However, existing KD methods do not explicitly distill the dependencies between labels, which limits the model ability to capture multi-label correlation. Furthermore, although existing methods for multi-label image classification have utilized the second-order label pair dependency (direct dependency between two labels), the high-order label pair dependency, which captures the indirect dependency between two labels, remains unexplored. In this paper, we propose a \textbf{\underline{M}}ulti-Order Label Pair \textbf{\underline{D}}ependencies \textbf{\underline{K}}nowledge \textbf{\underline{D}}istillation (MDKD) framework. MDKD explicitly distills the knowledge to capture multi-order dependencies between labels, including the label pair dependencies from second-order and high-order, thus transferring the insight of label correlations from different perspectives. Extensive experiments on Pascal VOC2007, MSCOCO2014, and NUS-WIDE demonstrate the superior performances of MDKD.

Primary Subject Area: [Content] Media Interpretation

Relevance To Conference: The Multi-Order Label Pair Dependencies Knowledge Distillation (MDKD) framework makes a significant contribution to multimedia/multimodal processing by addressing the challenge of multi-label image classification. Multi-label classification is essential for understanding and organizing multimedia content, where images often contain multiple objects that are interrelated. Traditional methods may not fully distill the dependencies between labels during knowledge distillation, leading to suboptimal performance. MDKD explicitly distills multi-order label pair dependencies, including both direct (second-order) and indirect (higher-order) dependencies, from a teacher network to a student network. This approach enables the student network to learn a more nuanced understanding of label correlations, reflecting both direct associations and complex indirect relationships between labels.

Submission Number: 2272