Empirical Bayes approach to Truth Discovery problems

Tsviel Ben Shabat, Reshef Meir, David Azriel

Keywords: Crowdsourcing, Empirical Bayes, Shrinkage, James–Stein estimator, Truth discovery

Abstract: When aggregating information from conflicting sources, one's goal is to find the truth. Most continuous-data Truth Discovery (TD) algorithms try to achieve this goal by estimating the trustworthiness of each source and then aggregating the conflicting information by weighing each source's answer proportionally to her trustworthiness. However, each of those algorithms requires more than a single source for such estimation and usually does not consider different estimation methods other than a weighted mean. Therefore, in this work we formulate, prove, and empirically test the conditions for an Empirical Bayes Estimator (EBE) to dominate the weighted mean aggregation. We further show that EBE is also a solution to the single source TD problem and we demonstrate that EBE, under mild conditions, can be used as a second step of any TD algorithm which can only improve the MSE.

TL;DR: Reducing the truth discovery problem to a single source and improving many existing truth discovery algorithms with a modified James-Stein Estimator.