In the last decade interest in the "wisdom of crowds" effect has gained momentum in both organizational research and corporate practice. Crowd wisdom relies on the aggregation of independent judgments. The accuracy of a group's aggregate prediction rises with the number, ability, and diversity of its members. We investigate these variables' relative importance for collective prediction using agent-based simulation. We replicate the "diversity trumps ability" proposition for large groups, showing that samples of heterogeneous agents outperform same-sized homogeneous teams of high ability. In groups smaller than about 16 members, however, the effects of group composition depend on the social decision function employed: Diversity is key only in continuous estimation tasks (averaging) and much less important in discrete choice tasks (voting), in which agents' individual abilities remain crucial. Thus, strategies to improve collective decision-making must adapt to the predictive situation at hand.
In our study, we have empirically studied the assessment of cited papers within the framework of the anchoring-and-adjustment heuristic. We are interested in the question whether the assessment of a paper can be influenced by numerical information that act as an anchor (e.g. citation impact). We have undertaken a survey of corresponding authors with an available email address in the Web of Science database. The authors were asked to assess the quality of papers that they cited in previous papers. Some authors were assigned to three treatment groups that receive further information alongside the cited paper: citation impact information, information on the publishing journal (journal impact factor) or a numerical access code to enter the survey. The control group did not receive any further numerical information. We are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation impact or journal impact), but also by numbers that are not related to quality, i.e. the access code. Our results show that the quality assessments of papers seem to depend on the citation impact information of single papers. The other information (anchors) such as an arbitrary number (an access code) and journal impact information did not play a (important) role in the assessments of papers. The results point to a possible anchoring bias caused by insufficient adjustment: it seems that the respondents assessed cited papers in another way when they observed paper impact values in the survey. We conclude that initiatives aiming at reducing the use of journal impact information in research evaluation either were already successful or overestimated the influence of this information.
The advantages of groups over individuals in complex decision-making have long interested scientists across disciplinary divisions. Averaging over a collection of individual judgments proves a reliable strategy for aggregating information, particularly in diverse groups in which statistically independent beliefs fall on both sides of the truth and contradictory biases are cancelled out. Social influence, some have said, narrows variation in individual opinions and undermines this wisdom-of-crowds effect in continuous estimation tasks. Researchers, however, neglected to study social-influence effects on voting in discrete choice tasks. Using agent-based simulation, we show that under voting — the most widespread social decision rule — social influence contributes to information aggregation and thus strengthens collective judgment. Adding to our knowledge about complex systems comprised of adaptive agents, this finding has important ramifications for the design of collective decision-making in both public administration and private firms.
In our planned study, we shall empirically study the assessment of cited papers within the framework of the anchoring-and-adjustment heuristic. We are interested in the question whether citation decisions are (mainly) driven by the quality of cited references. The design of our study is oriented towards the study by Teplitskiy, Duede [10]. We shall undertake a survey of corresponding authors with an available email address in the Web of Science database. The authors are asked to assess the quality of papers that they cited in previous papers. Some authors will be assigned to three treatment groups that receive further information alongside the cited paper: citation information, information on the publishing journal (journal impact factor), or a numerical access code to enter the survey. The control group will not receive any further numerical information. In the statistical analyses, we estimate how (strongly) the quality assessments of the cited papers are adjusted by the respondents to the anchor value (citation, journal, or access code). Thus, we are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation or journal), but also by numbers that are not related to quality, i.e. the access code. The results of the study may have important implications for quality assessments of papers by researchers and the role of numbers, citations, and journal metrics in assessment processes.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.