Crowdsourcing for Multiple-Choice Question Answering

Abstract

We leverage crowd wisdom for multiple-choice question answering, and employ lightweight machine learning techniques to improve the aggregation accuracy of crowdsourced answers to these questions. In order to develop more effective aggregation methods and evaluate them empirically, we developed and deployed a crowdsourced system for playing the “Who wants to be a millionaire?” quiz show. Analyzing our data (which consist of more than 200,000 answers), we find that by just going with the most selected answer in the aggregation, we can answer over 90% of the questions correctly, but the success rate of this technique plunges to 60% for the later/harder questions in the quiz show. To improve the success rates of these later/harder questions, we investigate novel weighted aggregation schemes for aggregating the answers obtained from the crowd. By using weights optimized for reliability of participants (derived from the participants’ confidence), we show that we can pull up the accuracy rate for the harder questions by 15%, and to overall 95% average accuracy. Our results provide a good case for the benefits of applying machine learning techniques for building more accurate crowdsourced question answering systems.

Learning from Noisy Crowd Labels with Logics

“…In addition, we analyze the inference performance of the truth inference methods on the training set. ( 1) On the sentiment polarity dataset, we consider the baseline MV, the two state-of-the-art graph models DS [3] and GLAD [4], and heuristic iterative inference based PM [10] and CATD [11]; (2) On the NER dataset, we consider the baseline MV, two NERoriented extended graph models DS [3] and IBCC [7] (which were originally designed for the traditional classification task), and ad hoc NER-oriented graph models BSC-seq [8] and HMM-Crowd [39].…”

Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective

“…SpectralDS [31], EBCC [32], [33], BayesDGC [34] MLNB [35], P-DS [36], ND-DS [36], MCMLD [37], MCMLD-OC [38] RY N [12] Discriminative MV, [39], KOS [40], [41], [42], PLAT [43], IEThresh [44] PV, [45], [46], CATD [47], PM [48], [49], [50], MNLDP [51], GTIC [52], [53], LLA [54], CrowdLayer [55], SpeeLFC [56] MLCC [57] Mean, Median CATD N [47], PM N [48] machine learning and data mining community first realized the opportunity that crowdsourcing brought to supervised learning, i.e., obtaining class labels for training sets. To improve the quality of labels, both Sheng et al [7] and Snow et al [8] proposed a repeated-labeling scheme in 2008, which let multiple crowd workers to label the same objects and the true labels of the objects are inferred from these multiple noisy labels.…”

“…Although MV is simple, it is very effective. Thus, researchers are still keen to study its variants [45], [39], [46]. For example, Tao et al [46] proposed four strategies to model the similarity of crowdsourced labels.…”

An Agricultural Harvest Knowledge Survey to Distinguish Types of Expertise

“…The questions were presented to the operator in the form of a web survey via Qualtrics as seen in Figure 1. To score the harvest knowledge survey, the top two answers were identified from all answers submitted in a "wisdom of the crowd" type evaluation (Aydin, Yilmaz, Li, & Li, 2014;Yi, Steyvers, Lee, & Dry, 2012). The top two answers were validated with expert engineers, combine performance software (Deere, 2012(Deere, , 2013b, and the John Deere field adjustment guide (Deere, 2013a).…”