Domain-Weighted Majority Voting for Crowdsourcing

Abstract

Crowdsourcing labeling systems provide an efficient way to generate multiple inaccurate labels for given observations. If the competence level or the ``reputation,'' which can be explained as the probabilities of annotating the right label, for each crowdsourcing annotators is equal and biased to annotate the right label, majority voting (MV) is the optimal decision rule for merging the multiple labels into a single reliable one. However, in practice, the competence levels of annotators employed by the crowdsourcing labeling systems are often diverse very much. In these cases, weighted MV is more preferred. The weights should be determined by the competence levels. However, since the annotators are anonymous and the ground-truth labels are usually unknown, it is hard to compute the competence levels of the annotators directly. In this paper, we propose to learn the weights for weighted MV by exploiting the expertise of annotators. Specifically, we model the domain knowledge of different annotators with different distributions and treat the crowdsourcing problem as a domain adaptation problem. The annotators provide labels to the source domains and the target domain is assumed to be associated with the ground-truth labels. The weights are obtained by matching the source domains with the target domain. Although the target-domain labels are unknown, we prove that they could be estimated under mild conditions. Both theoretical and empirical analyses verify the effectiveness of the proposed method. Large performance gains are shown for specific data sets.

Active Multilabel Crowd Consensus

“…Since workers have diverse qualities on different tasks, a better task assignment strategy may also contribute to a better consensus. Many researches on improving task design have been explored from different perspectives, such as lower complexity [41], worker's expertise [42], [43], checking workers' answers [44], and so on. To name a few, Zhang et al [42] considered an expertise-aware task allocation problem in mobile crowdsourcing, where the worker's expertise is obtained based on semantic analysis.…”

Weighted ensemble networks formultiviewbased tiny object quality assessment

“…Compared with single model learning, the main advantage of ensemble learning is the improved generalization ability and the flexible mapping between individual systems. 27,28 Generally, the ensemble learning methods merge the features from different sub-models into a unified feature representation. 29 Bagging, 30 boosting, 31 and stacking 32 are three typical ensemble learning strategies.…”

CrowdWT

“…Some other solutions assume workers have different levels of skills depending on the domain. As such, they partition tasks into domains, and assign tasks to workers whose skills are a good fit [33,49].…”

“…The worker-centered strategy typically assumes a set of workers with known qualities, it evaluates a worker's preference on different tasks, and assigns the tasks to workers with high qualities. However, it often ignores the specific requirements of a task [33,49]. Some recent studies account for both tasks and workers; they model the difference between worker's skills and task difficulty [17,48].…”