Multi-label Answer Aggregation Based on Joint Matrix Factorization

IEEE Trans. Neural Netw. Learning Syst.

et al. 2021

Self Cite

Crowdsourcing is an economic and efficient strategy aimed at collecting annotations of data through an online platform. Crowd workers with different expertise are paid for their service, and the task requester usually has a limited budget. How to collect reliable annotations for multi-label data and how to compute the consensus within budget is an interesting and challenging, but rarely studied, problem.In this paper, we propose a novel approach to accomplish Active Multi-label Crowd Consensus (AMCC). AMCC accounts for the commonality and individuality of workers, and assumes that workers can be organized into different groups. Each group includes a set of workers who share a similar annotation behavior and label correlations. To achieve an effective multilabel consensus, AMCC models workers' annotations via a linear combination of commonality and individuality, and reduces the impact of unreliable workers by assigning smaller weights to the group. To collect reliable annotations with reduced cost, AMCC introduces an active crowdsourcing learning strategy that selects sample-label-worker triplets. In a triplet, the selected sample and label are the most informative for the consensus model, and the selected worker can reliably annotate the sample with low cost. Our experimental results on multi-label datasets demonstrate the advantages of AMCC over state-of-the-art solutions on computing crowd consensus and on reducing the budget by choosing cost-effective triplets.

Section: Introductionmentioning

confidence: 53%

Active Multilabel Crowd Consensus

IEEE Trans. Neural Netw. Learning Syst.

et al. 2021

Self Cite

“…To avoid this limitation, researchers have resorted to develop semi-supervised multilabel classifiers [23], [24], [25], [26], [27], in which limited labeled samples as well as abundant unlabeled samples are jointly used for training. Besides, considering the fact that labeled data is tagged by human efforts, they might have some missing or noisly labels [28], [29], [30], [31], [32], [33], several approaches have been proposed to design multi-label classifiers under weak-label setting [28], [29], [34], [35] or with noisy labels [36], [32], [37], [38].…”

Section: A Multi-label Learningmentioning

confidence: 99%

Individuality- and Commonality-Based Multiview Multilabel Learning

Tan

et al. 2021

IEEE Trans. Cybern.

Self Cite

In multi-view multi-label learning, each object is represented by several heterogeneous feature representations and is also annotated with a set of discrete non-exclusive labels. Previous studies typically focus on capturing the shared latent patterns among multiple views, while do not sufficiently consider the diverse characteristics of individual views, which can cause performance degradation. In this paper, we propose a novel approach (ICM2L) to explicitly explore the individuality and commonality information of multi-label multiple view data in a unified model. Specifically, a common subspace is learned across different views to capture the shared patterns. Then, multiple individual classifiers are exploited to explore the characteristics of individual views. Next, an ensemble strategy is adopted to make prediction. Finally, we develop an alternative solution to jointly optimize our model, which can enhance the robustness of the proposed model towards rare labels and reinforce the reciprocal effects of individuality and commonality among heterogeneous views, and thus further improve the performance. Experiments on various real-word datasets validate the effectiveness of ICM2L against state-of-the-art solutions, and ICM2L can leverage the individuality and commonality information to achieve an improved performance as well as to enhance the robustness toward rare labels.

“…Improving data quality. The most intuitive strategy to deal with the low quality of data annotation is to improve the quality of the data itself [12,5,13,14,16,17]. The Dawid-Skene (DS) model [12] is a standard probabilistic model for label inference from multiple annotations using Expectation-Maximization (EM).…”

Section: Related Workmentioning

confidence: 99%

“…AWMV uses the frequency of positive labels in multiple noisy label sets of each task to estimate a bias rate, and then assigns weights, derived from the bias rate, to negative and positive labels. These methods tackle the problem of identifying low quality answers in different ways [12,5,14,16,13,17]. However, they all ignore the intrinsic features of tasks, which can improve the quality of aggregated labels.…”

Section: Related Workmentioning

confidence: 99%

Attention-Aware Answers of the Crowd

Tu¹,

Proceedings of the 2020 SIAM International Conference on Data Mining

et al. 2020

Self Cite

Crowdsourcing is a relatively economic and efficient solution to collect annotations from the crowd through online platforms. Answers collected from workers with different expertise may be noisy and unreliable, and the quality of annotated data needs to be further maintained. Various solutions have been attempted to obtain high-quality annotations. However, they all assume that workers' label quality is stable over time (always at the same level whenever they conduct the tasks). In practice, workers' attention level changes over time, and the ignorance of which can affect the reliability of the annotations. In this paper, we focus on a novel and realistic crowdsourcing scenario involving attention-aware annotations. We propose a new probabilistic model that takes into account workers' attention to estimate the label quality. Expectation propagation is adopted for efficient Bayesian inference of our model, and a generalized Expectation Maximization algorithm is derived to estimate both the ground truth of all tasks and the label-quality of each individual crowd worker with attention. In addition, the number of tasks best suited for a worker is estimated according to changes in attention. Experiments against related methods on three realworld and one semi-simulated datasets demonstrate that our method quantifies the relationship between workers' attention and label-quality on the given tasks, and improves the aggregated labels. 4 In this paper, the definition of attention is the ability to focus on finishing something. We study the process of the effect of attention on answering crowdsourcing tasks, rather than the process of attention being influenced by certain factors (e.g., fatigue, frustration, or some combination of other concepts).