Crowdsourced research: Many hands make tight work

Silberzahn, Raphael; Uhlmann, Eric Luis

doi:10.1038/526189a

Cited by 161 publications

(108 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ensuring that the relationships identified in an empirical study are trustworthy is important in IB studies in particular and in management studies in general, because there is a very limited tradition of replicability that can help uncover researchers' biases and differences in empirical techniques (Bettis, Ethiraj, Gambardella, Helfat, & Mitchell, 2016;Silberzahn & Uhlmann, 2015). This limited replicability is due to several reasons.…”

Section: Introductionmentioning

confidence: 99%

From the Editors: Can I trust your findings? Ruling out alternative explanations in international business research

et al. 2016

View full text Add to dashboard Cite

The complex nature of international business research, with its cross-country and multilevel nature, complicates the empirical identification of relationships among theoretical constructs. The objective of this editorial is to provide guidance to help international business scholars navigate this complexity and ensure that readers can trust their findings. We provide suggestions for how to rule out alternative explanations, explaining key considerations not only in empirical analyses, but also in theory building and in research design. Our discussion covers both qualitative and quantitative studies, because we believe that it is imperative to understand how trustworthiness is established in both traditions, even for international business researchers who self-identify with only one. This enables scholars to have a broader scope of knowledge when interpreting past research in the field and to be more adept at explaining their design choices to a diverse audience.

show abstract

Section: Introductionmentioning

confidence: 99%

From the Editors: Can I trust your findings? Ruling out alternative explanations in international business research

et al. 2016

View full text Add to dashboard Cite

show abstract

“…In fact, the multiplicity of possible data analysis approaches is even an issue on its own as recently illustrated by Silberzahn and Uhlmann (2015). Whereas such "embarras du choix," related to the "multiplicity of perspectives"-including but not limited to model selection criteria-described by Gelman and Hennig (2017), is not bad per se and one should not attempt to eliminate it by formulating strict guidelines, it is not clear how one should deal with multiple approaches in practice.…”

mentioning

confidence: 99%

On the necessity and design of studies comparing statistical methods

et al. 2017

View full text Add to dashboard Cite

In data analysis sciences in general and in biometrical research particularly, there are strong incentives for presenting work that entails new methods. Many journals require authors to propose new methods as a prerequisite for publication, as this is the most straightforward way to claim the necessary novelty. The development of new methods is also factually often a sine qua non condition to be recruited as a faculty member or to obtain personnel funding from a methods-oriented research agency, not least because it noticeably increases the chance to get published as outlined above. Thus, in statistical research and related methodology-oriented fields such as machine learning or bioinformatics, the well-known adage "publish or perish" could be translated into "propose new methods or perish."Such a research paradigm is not favorable for studies that aim at meaningfully comparing alternative existing methods or, more generally, studies assessing the behavior and properties of existing methods. Yet, given the exponential increase in the number and complexity of new statistical methods being published every year, the end users are often at a loss regarding what are the "optimal" or even "appropriate" methods to answer the research question of interest given a particular data structure. It becomes more and more difficult to get an overview of existing methods, not to mention the overview of their respective performances in different settings (Sauerbrei, Abrahamowicz, Altman, Le Cessie, & Carpenter, 2014).Moreover, it is well known that studies comparing a suggested new method to existing methods may be (strongly) biased in favor of the new method. This is a consequence of various factors starting with the authors' better expertise on the new method compared to the competing methods. Another factor is the combination of publication pressure (publish or perish) and publication bias-in the sense that a new method performing worse than existing ones has (severe) difficulties to get published (Boulesteix, Stierle, & Hapfelmeier, 2015). This may lead to simulation designs that might be-intentionally or unintentionallybiased. Note that not only empirical evaluations but also theoretical properties suggesting the superiority of a method under particular assumptions may be in principle potentially affected by this kind of bias. Deriving theoretical results for statistical approaches relevant in practice is extremely difficult and possible only under strong assumptions (Picard & Cook, 1984). We speculate that authors assessing the theoretical properties of their new method tend to make assumptions that are rather favorable for the new method-also a form of bias.In contrast, neutral comparison studies, as defined by Boulesteix, Wilson, and Hapfelmeier (2017a), are dedicated to the comparison itself: they do not aim to demonstrate the superiority of a particular method and are thus not designed in a way that may increase the probability to observe incorrectly this superiority. Furthermore, they involve authors who are, as a coll...

show abstract

“…The publicly available dataset described in this paper was collected through crowdsourcing 14 during the period from December 2013 to October 2016 from 1,144 participants aged between 18 and 89 with various educational and cultural backgrounds. All study procedures were conducted in English.…”

Section: Background and Summarymentioning

confidence: 99%

A dataset of human decision-making in teamwork management

Shen

Miao

et al. 2017

Sci Data

View full text Add to dashboard Cite

Today, most endeavours require teamwork by people with diverse skills and characteristics. In managing teamwork, decisions are often made under uncertainty and resource constraints. The strategies and the effectiveness of the strategies different people adopt to manage teamwork under different situations have not yet been fully explored, partially due to a lack of detailed large-scale data. In this paper, we describe a multi-faceted large-scale dataset to bridge this gap. It is derived from a game simulating complex project management processes. It presents the participants with different conditions in terms of team members’ capabilities and task characteristics for them to exhibit their decision-making strategies. The dataset contains detailed data reflecting the decision situations, decision strategies, decision outcomes, and the emotional responses of 1,144 participants from diverse backgrounds. To our knowledge, this is the first dataset simultaneously covering these four facets of decision-making. With repeated measurements, the dataset may help establish baseline variability of decision-making in teamwork management, leading to more realistic decision theoretic models and more effective decision support approaches.

show abstract

Crowdsourced research: Many hands make tight work

Cited by 161 publications

References 3 publications

From the Editors: Can I trust your findings? Ruling out alternative explanations in international business research

From the Editors: Can I trust your findings? Ruling out alternative explanations in international business research

On the necessity and design of studies comparing statistical methods

A dataset of human decision-making in teamwork management

Contact Info

Product

Resources

About