Spectral Clustering and Labeling for Crowdsourcing with Inherently Distinct Task Types

TMLR Paper4271 Authors

Abstract: The Dawid-Skene model is the most widely assumed model in the analysis of crowdsourcing algorithms that estimate ground-truth labels from noisy worker responses. In this work, we are motivated by crowdsourcing applications where workers have distinct skill sets and their accuracy additionally depends on a task's type. Focusing on the case where there are two types of tasks, we propose a spectral method to partition tasks into two groups such that a worker has the same reliability for all tasks within a group. Our analysis reveals a separability condition such that task types can be perfectly recovered if the number of workers $n$ scales logarithmically with the number of tasks $d$. Numerical experiments show how clustering tasks by type before estimating ground-truth labels enhances the performance of crowdsourcing algorithms in practical applications.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

This is a revised version of the manuscript under submission (Paper 4271). We have made several updates based on the reviewers' comments. The main changes are summarized below:

**Expanded Related Work:** We have significantly extended the related work section to better position our contributions within the existing literature. We now cite and compare against the specific papers mentioned by the reviewers, and provide a broader discussion of recent work in crowdsourcing with multi-type models. We also clarify how our analysis differs from classical spectral clustering results in the stochastic block model (SBM) and Gaussian mixture model literatures, and why those techniques are not directly applicable to our setting.

**Clarified Novelty and Contributions:** Added a note in the contributions sections to more clearly articulate the novelty and significance of our results by comparing with the classical spectral clustering results.

**Notation Table:** A table of key notations has been added to the main paper to improve readability and accessibility.

**Geometric Interpretation:** We added a geometric interpretation of the clustering theorem (Remark 1) to enhance conceptual understanding.

**Runtime Comparison:** We now include a runtime comparison of the different algorithms, detailing both the theoretical complexity and empirical execution time, including the overhead introduced by the clustering step.

**Synthetic Experiment:** A new synthetic experiment has been added (Appendix C.4) to illustrate how clustering error varies with the number of workers and the difference between reliability vectors.

**Broader Impact Statement:** We have elaborated on and expanded the discussion in the broader impact section.

Assigned Action Editor: ~Jinwoo_Shin1

Submission Number: 4271