Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences

Köhler, Tine; Cortina, José M.

doi:10.1177/0149206319843985

Cited by 137 publications

(201 citation statements)

References 97 publications

Supporting

Mentioning

174

Contrasting

Unclassified

Order By: Relevance

“…After an initial coding attempt with ill-defined coding criteria had led to too many disagreements (described further in the Appendix), we developed the following strategy (not pre-registered): We searched the full texts of all papers for the string 'replic * ' (cf. Makel, Plucker, & Hegarty, 2012;Köhler & Cortina, 2019;Mueller-Langer, Fecher, Harhoff, & Wagner, 2019;Pridemore, Makel, & Plucker, 2018) and, for papers that did contain it, determined whether the coded hypothesis was a close replication with the goal to verify a previously published result. Conceptual replications and internal replications (replication of a study in the same paper) were not counted as replications in this narrow sense, since both are more likely to be motivated by the goal to build on previous work than by scepticism.…”

Section: Measures and Coding Proceduresmentioning

confidence: 99%

An excess of positive results: Comparing the standard Psychology literature with Registered Reports

Scheel¹,

Schijen²,

Lakens³

2020

Preprint

135

View full text Add to dashboard Cite

When studies with positive results that support the tested hypotheses have a higher probability of being published than studies with negative results, the literature will give a distorted view of the evidence for scientific claims. Psychological scientists have been concerned about the degree of distortion in their literature due to publication bias and inflated Type-1 error rates. Registered Reports were developed with the goal to minimise such biases: In this new publication format, peer review and the decision to publish take place before the study results are known. We compared the results in the full population of published Registered Reports in Psychology (N = 71 as of November 2018) with a random sample of hypothesis-testing studies from the standard literature (N = 152) by searching 633 journals for the phrase ‘test* the hypothes*’ (replicating a method by Fanelli, 2010). Analysing the first hypothesis reported in each paper, we found 96% positive results in standard reports, but only 44% positive results in Registered Reports. The difference remained nearly as large when direct replications were excluded from the analysis (96% vs 50% positive results). This large gap suggests that psychologists underreport negative results to an extent that threatens cumulative science. Although our study did not directly test the effectiveness of Registered Reports at reducing bias, these results show that the introduction of Registered Reports has led to a much larger proportion of negative results appearing in the published literature compared to standard reports.

show abstract

Section: Measures and Coding Proceduresmentioning

confidence: 99%

An excess of positive results: Comparing the standard Psychology literature with Registered Reports

Scheel¹,

Schijen²,

Lakens³

2020

Preprint

135

View full text Add to dashboard Cite

show abstract

“…One of them is how to define the universe of all possible operationalizations of a concept (classical operationalism limits the meaning of a concept to established operations), which is actually a problem more intractable than it first appears to be. For example, it might not be ideal to include a measure that is known for its poor psychometric qualities in that universe just because of its connection to the concept (Köhler & Cortina, 2021). Or we can always (and often do) imagine that future researchers will come up with a much better, previously unthought of measure of a concept that would clearly win out over its existing alternatives (you may think of Popper's black swan in terms of measurement).…”

Section: On Random Effects)mentioning

confidence: 99%

A Falsificationist Treatment of Auxiliary Hypotheses in Social and Behavioral Sciences: Systematic Replications Framework

Tunç¹,

Mn²

2020

Preprint

View full text Add to dashboard Cite

Single empirical tests are always ambiguous in their implications for the theory under investigation, because non-corroborative evidence leaves us underdetermined in our decision as to whether the main hypothesis or one or more auxiliary hypotheses should bear the burden of falsification. Methodological falsificationism tries to solve this problem by relegating auxiliary hypotheses that increase the testability of theories to unproblematic background knowledge and disallowing others. However, decisions to accept such auxiliaries as unproblematic are seldom conclusively justified in the social and behavioral sciences, where operationalizations play a central role, but are much less theory-driven and independently testable. Close and conceptual replications are crucial in tackling different aspects of underdetermination, but they fail to serve this purpose when conducted in isolation. To facilitate rational decision-making regarding falsifications, we propose Systematic Replications Framework (SRF) that organizes subsequent tests into a pre-planned series of logically interlinked close and conceptual replications. SRF reduces underdetermination by disentangling the implications of non-corroborative findings for the main hypothesis and the operationalization-related auxiliaries. It also serves as a severe-testing procedure through systematically organized self-replications. We further discuss how SRF will be particularly useful if applied to contested theoretical claims with mixed evidence and realized through adversarial collaboration. We argue that SRF will facilitate establishment of scientific consensus regarding established evidence and how exactly it supports or contradicts competing theoretical claims, which will in turn allow us to more objectively appraise scientific progress in the social and behavioral sciences.

show abstract

“…Don Hambrick, one of the pioneers of UET, recently argued that these conditions present unique opportunities for researchers: to advise leaders on how to confront a shifting environment and to leverage our existing UET research platform to take on broader questions (Hambrick, 2019). Yet fundamental issues are also now being raised regarding the transparency, replicability, and applicability of social science research, including management research (e.g., Köhler & Cortina, in press; Rynes, Colbert, & O’Boyle, 2018). So while the relevance of the UET perspective may be increasing, so are the challenges of executing sound studies—underscoring the importance of addressing the metacritiques we examine in this review.…”

mentioning

confidence: 99%

Metacritiques of Upper Echelons Theory: Verdicts and Recommendations for Future Research

Neely

Lovelace²,

Cowen

et al. 2020

Journal of Management

182

174

View full text Add to dashboard Cite

After more than 35 years, Hambrick and Mason’s upper echelons theory (UET) stands as one of the most influential perspectives in management research. However, as the literature and its attendant reviews have become more numerous and specialized, discussion of the fundamental conceptual and methodological critiques leveled against research utilizing the UET perspective has grown fragmented. As such, the first aim of the present review is to identify and synthesize a set of common critiques levied against UET research. In doing so, we unpack important nuance within each critique while establishing a common vocabulary to facilitate greater consistency in how these critiques are discussed by scholars, editors, and reviewers. Next, we analyze the past decade of UET research (i.e., empirical and conceptual articles) to provide a progress report on the state of the literature as it pertains to each critique. Based on this analysis, we issue a “verdict,” providing clear guidance on what issues still need to be resolved and offering clear recommendations for UET scholars moving forward. Finally, we close by taking stock of how our recommendations position UET scholars to embrace the task of providing impactful insights on emerging challenges facing top executives.

show abstract

Play It Again, Sam! An Analysis of Constructive Replication in the Organizational Sciences

Abstract: We wish to express our gratitude to Fred Oswald and two anonymous reviewers for their insightful comments. Work on the article was supported by the University of Melbourne's Faculty of Business and Economics.

Cited by 137 publications

References 97 publications

An excess of positive results: Comparing the standard Psychology literature with Registered Reports

An excess of positive results: Comparing the standard Psychology literature with Registered Reports

A Falsificationist Treatment of Auxiliary Hypotheses in Social and Behavioral Sciences: Systematic Replications Framework

Metacritiques of Upper Echelons Theory: Verdicts and Recommendations for Future Research

Contact Info

Product

Resources

About