[PDF][PDF] Crowdsourcing label quality: a theoretical analysis.

W Wang, ZH Zhou - Sci. China Inf. Sci., 2015 - lamda.nju.edu.cn
Sci. China Inf. Sci., 2015lamda.nju.edu.cn
Crowdsourcing has been an effective and efficient paradigm for providing labels for large-
scale unlabeled data. In the past few years, many methods have been developed for
inferring labels from the crowd, but few theoretical analyses have been presented to support
this popular human-machine interaction process. In this paper, we theoretically study the
quality of labels inferred from crowd workers by majority voting and provide an analysis of
label quality that shows that the label error rate decreases exponentially with the number of …
Abstract
Crowdsourcing has been an effective and efficient paradigm for providing labels for large-scale unlabeled data. In the past few years, many methods have been developed for inferring labels from the crowd, but few theoretical analyses have been presented to support this popular human-machine interaction process. In this paper, we theoretically study the quality of labels inferred from crowd workers by majority voting and provide an analysis of label quality that shows that the label error rate decreases exponentially with the number of workers selected for each task. We also study the problem of eliminating low-quality workers from the crowd, and provide a conservative condition for eliminating low-quality workers without eliminating any non-low-quality worker with high probability. We also provide an aggressive condition for eliminating all low-quality workers with high probability.
lamda.nju.edu.cn
Showing the best result for this search. See all results