Wisdom of crowds and collective decision-making in a survival situation with complex information integration

Hamada, Daisuke; Nakayama, Masataka; Saiki, Jun

doi:10.1186/s41235-020-00248-z

“…The second limitation is the method of aggregating individuals' judgments. Although only one majority rule was applied in this study, many previous studies on the wisdom of crowds have discussed different ways to effectively aggregate individuals' judgments for improving group judgments [6], [30][31][32] . Some studies have argued the importance of considering subjective con dence in group judgments [30], [32 .…”

Section: Discussionmentioning

confidence: 99%

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

Shirasuna

¹

,

Honda

²

2021

Preprint

0

View full text Add to dashboard Cite

In group judgments in a binary choice task, the judgments of individuals with low confidence (i.e., they feel that the judgment was not correct) may be regarded as unreliable. Previous studies have shown that aggregating individuals’ diverse judgments can lead to high accuracy in group judgments, a phenomenon known as the wisdom of crowds. Therefore, if low-confidence individuals make diverse judgments between individuals and the mean of accuracy of their judgments is above the chance level (.50), it is likely that they will not always decrease the accuracy of group judgments. To investigate this issue, the present study conducted behavioral experiments using binary choice inferential tasks, and computer simulations of group judgments by manipulating group sizes and individuals’ confidence levels. Results revealed that (I) judgment patterns were highly similar between individuals regardless of their confidence levels; (II) the low-confidence group could make judgments as accurate as the high-confidence group, as the group size increased; and (III) even if there were low-confidence individuals in a group, they generally did not inhibit group judgment accuracy. The results suggest the usefulness of low-confidence individuals’ judgments in a group and provide practical implications for real-world group judgments.

show abstract

“…The second limitation is the method of aggregating individuals' judgments. Although only one majority rule was applied in this study, many previous studies on the wisdom of crowds have discussed different ways to effectively aggregate individuals' judgments for improving group judgments [6], [30][31][32] . Some studies have argued the importance of considering subjective confidence in group judgments [30], [32 .…”

Section: Discussionmentioning

confidence: 99%

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

Shirasuna¹,

Honda²

2021

Preprint

0

View full text Add to dashboard Cite

In group judgments in a binary choice task, the judgments of individuals with low confidence (i.e., they feel that the judgment was not correct) may be regarded as unreliable. Previous studies have shown that aggregating individuals’ diverse judgments can lead to high accuracy in group judgments, a phenomenon known as the wisdom of crowds. Therefore, if low-confidence individuals make diverse judgments between individuals and the mean of accuracy of their judgments is above the chance level (.50), it is likely that they will not always decrease the accuracy of group judgments. To investigate this issue, the present study conducted behavioral experiments using binary choice inferential tasks, and computer simulations of group judgments by manipulating group sizes and individuals’ confidence levels. Results revealed that (I) judgment patterns were highly similar between individuals regardless of their confidence levels; (II) the low-confidence group could make judgments as accurate as the high-confidence group, as the group size increased; and (III) even if there were low-confidence individuals in a group, they generally did not inhibit group judgment accuracy. The results suggest the usefulness of low-confidence individuals’ judgments in a group and provide practical implications for real-world group judgments.

show abstract

“…In the context of group decision-making, Grofman et al ( 1983 ) suggested weighing each individual's inputs based on self-reported confidence of their respective responses, in accordance with the belief that individuals can estimate reliably the accuracy of their own judgments (Griffin and Tversky, 1992 ). More recently, Hamada et al ( 2020 ) designed a wisdom of the crowds study that asked a set of participants to rank and rate 15 items they would need for survival and used weighted confidence values to aggregate their inputs. The results were sensitive to the size of the group (i.e., number of participants); when the group was small (fewer than 10 participants), the confidence values reportedly had little impact on the results.…”

Section: Literature Reviewmentioning

confidence: 99%

Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

Yasmin

¹

,

Hassan²,

Grassel³

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

This work investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Five types of input elicitation methods are tested: binary classification (positive or negative); the (x, y)-coordinate of the position participants believe a target object is located; level of confidence in binary response (on a scale from 0 to 100%); what participants believe the majority of the other participants' binary classification is; and participant's perceived difficulty level of the task (on a discrete scale). We design two crowdsourcing studies to test the performance of a variety of input elicitation methods and utilize data from over 300 participants. Various existing voting and machine learning (ML) methods are applied to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experiment results suggest that more accurate results can be achieved with smaller training datasets when both the crowdsourced binary classification labels and the average of the self-reported confidence values in these labels are used as features for the ML classifiers. Moreover, when a relatively larger properly annotated dataset is available, in some cases augmenting these ML algorithms with the results (i.e., probability of outcome) from an automated classifier can achieve even higher performance than what can be obtained by using any one of the individual classifiers. Lastly, supplementary analysis of the collected data demonstrates that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods.

show abstract

Wisdom of crowds and collective decision-making in a survival situation with complex information integration

Cited by 23 publications

References 38 publications

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

Do low-confidence individuals decrease group judgments’ accuracy? Investigations in terms of the wisdom of crowds framework

Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

Contact Info

Product

Resources

About