Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective

Abstract

Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily involved to facilitate the knowledge learning process. During the past thirteen years, researchers in the AI community made great efforts to remove the obstacles in the field of learning from crowds. This concentrated survey paper comprehensively reviews the technical progress in crowdsourcing learning from a systematic perspective that includes three dimensions of data, models, and learning processes. In addition to reviewing existing important work, the paper places a particular emphasis on providing some promising blueprints on each dimension as well as discussing the lessons learned from our past research work, which will light up the way for new researchers and encourage them to pursue new contributions.

Performance and Agreement When Annotating Chest X-ray Text Reports—A Preliminary Step in the Development of a Deep Learning-Based Prioritization and Detection System

“…The results of our research indicated that there could be a reduction in false positive labels when using majority labeling compared to the labels used by an individual annotator ( Table A3 ). Recent efforts have been made to outsource labeling to more annotators of lesser specialized experience as a way to reduce the time and cost of data gathering compared to sourcing and reimbursing field experts in the same tasks [ 44 ]. Several methods have been proposed to clean data labeled by multiple, less experienced annotators to obtain high-quality datasets efficiently, including using majority-vote labeling [ 45 , 46 , 47 ].…”

Classifying Workers for Mitigating Adversarial Attacks in Crowdsourcing

“…However, it is very prone to adversarial attacks and does not ensure the quality of crowdsourced answers. There are several methods for inferring the truth of tasks and for performing aggregation of submissions, such as [9,[15][16][17][18]. They are classified based on the underlying techniques as direct computation [15], probability-based methods [19], optimization methods [20], and neural network-based (NN) methods [21].…”

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

“…However, all these methods do not have the advantages of the recently proposed neuralized HMM-based graphical models [18,19] and our Neural-Hidden-CRF in principled modeling for variants of interest and in harnessing the context information that provided by advanced deep learning models. Additionally, it is worth mentioning the presence of numerous established WS methods that address the normal independent classification scenario [3,5,[43][44][45].…”