A Square Peg in a Square Hole: Meta-Expert for Long-Tailed Semi-Supervised Learning

Yaxin Hou, Yuheng Jia

TL;DR: We propose a novel long-tailed semi-supervised learning with distribution mismatch method called Meta-Expert.

Abstract: This paper studies the long-tailed semi-supervised learning (LTSSL) with distribution mismatch, where the class distribution of the labeled training data follows a long-tailed distribution and mismatches with that of the unlabeled training data. Most existing methods introduce auxiliary classifiers (experts) to model various unlabeled data distributions and produce pseudo-labels, but the expertises of various experts are not fully utilized. We observe that different experts are good at predicting different intervals of samples, e.g., long-tailed expert is skilled in samples located in the head interval and uniform expert excels in samples located in the medium interval. Therefore, we propose a dynamic expert assignment module that can estimate the class membership (i.e., head, medium, or tail class) of samples, and dynamically assigns suitable expert to each sample based on the estimated membership to produce high-quality pseudo-label in the training phase and produce prediction in the testing phase. We also theoretically reveal that integrating different experts' strengths will lead to a smaller generalization error bound. Moreover, we find that the deeper features are more biased toward the head class but with more discriminative ability, while the shallower features are less biased but also with less discriminative ability. We, therefore, propose a multi-depth feature fusion module to utilize different depth features to mitigate the model bias. Our method demonstrates its effectiveness through comprehensive experiments on the CIFAR-10-LT, STL-10-LT, and SVHN-LT datasets across various settings. The code is available at https://github.com/yaxinhou/Meta-Expert.

Lay Summary:

In many real-world applications, we often have limited labeled data, and these labeled examples are imbalanced — some classes have many samples (like “cats”), while others have very few (like “hamsters”). At the same time, we may have a large amount of unlabeled data with a different class distribution. This situation makes training accurate AI models especially difficult.

Our research focuses on improving how AI learns in this challenging setting. We found that different expert models are good at labeling different kinds of data — some are better with common classes, while others handle rare classes more accurately. So, we designed a system that first estimates whether a piece of data belongs to a common, medium, or rare class, and then selects the most suitable expert model to label it. This improves the quality of the AI’s learning process.

We also discovered that information from different layers of the AI model behaves differently: shallow layers are more balanced, while deeper layers are more powerful but more biased. To address this, we combine information from multiple layers to get the best of both worlds.

Our method achieves better performance on several benchmark datasets and provides a more reliable way to train AI models with limited and imbalanced data.

Link To Code: https://github.com/yaxinhou/Meta-Expert

Primary Area: General Machine Learning->Unsupervised and Semi-supervised Learning

Keywords: Long tailed learning, semi-supervised learning

Submission Number: 6085