Realistic Evaluation of Deep Partial-Label Learning Algorithms
Keywords: Partial-label learning, weakly supervised learning, benchmark.
TL;DR: The first partial-label learning benchmark with a new dataset of human-annotated partial labels.
Abstract: Partial-label learning (PLL) is a weakly supervised learning problem in which
each example is associated with multiple candidate labels and only one is the
true label. In recent years, many deep PLL algorithms have been developed to
improve model performance. However, we find that some early developed
algorithms are often underestimated and can outperform many later algorithms
with complicated designs. In this paper, we delve into the empirical
perspective of PLL and identify several critical but previously overlooked
issues. First, model selection for PLL is non-trivial, but has never been
systematically studied. Second, the experimental settings are highly
inconsistent, making it difficult to evaluate the effectiveness of the
algorithms. Third, there is a lack of real-world image datasets that can be
compatible with modern network architectures. Based on these findings, we
propose PLENCH, the first Partial-Label learning bENCHmark to systematically
compare state-of-the-art deep PLL algorithms. We investigate the model
selection problem for PLL for the first time, and propose novel model selection
criteria with theoretical guarantees. We also create Partial-Label CIFAR-10
(PLCIFAR10), an image dataset of human-annotated partial labels collected from
Amazon Mechanical Turk, to provide a testbed for evaluating the performance of
PLL algorithms in more realistic scenarios. Researchers can quickly and
conveniently perform a comprehensive and fair evaluation and verify the
effectiveness of newly developed algorithms based on PLENCH. We hope that
PLENCH will facilitate standardized, fair, and practical evaluation of PLL
algorithms in the future.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5419
Loading