Merlin L48 Spectrogram Dataset

Aaron Sun, Subhransu Maji, Grant Van Horn

Keywords: multilabel, semi-supervised, spectrogram, dataset, single-positive, fine-grained

TL;DR: Realistic fine-grained multi-label dataset with evaluation and analysis on SPML methods in multiple data regimes.

Abstract: In the single-positive multi-label (SPML) setting, each image in a dataset is labeled with the presence of a single class, while the true presence of other classes remains unknown. The challenge is to narrow the performance gap between this partially-labeled setting and fully-supervised learning, which often requires a significant annotation budget. Prior SPML methods were developed and benchmarked on synthetic datasets created by randomly sampling single positive labels from fully-annotated datasets like Pascal VOC, COCO, NUS-WIDE, and CUB200. However, this synthetic approach does not reflect real-world scenarios and fails to capture the fine-grained complexities that can lead to difficult misclassifications. In this work, we introduce the L48 dataset, a fine-grained, real-world multi-label dataset derived from recordings of bird sounds. L48 provides a natural SPML setting with single-positive annotations on a challenging, fine-grained domain, as well as two extended settings in which domain priors give access to additional negative labels. We benchmark existing SPML methods on L48 and observe significant performance differences compared to synthetic datasets and analyze method weaknesses, underscoring the need for more realistic and difficult benchmarks.

Dataset URL: https://msid-ml48s.s3.amazonaws.com/v0/ml48s.tar.gz

Code URL: https://github.com/cvl-umass/l48-benchmarking

Primary Area: Datasets & Benchmarks for applications in computer vision

Submission Number: 266