A Replication Study of Operations Management Experiments in Management Science

Davis, Andrew M.; Flicker, Blair; Hyndman, Kyle; Katok, Elena; Keppler, Samantha; Leider, Stephen; Long, Xiaoyang; Tong, Jonathan

doi:10.2139/ssrn.4135500

Cited by 5 publications

(9 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overall, we analyze data for 759 firms over time (11,463 data points). Notable studies have implemented similar large‐scale RCTs across contexts (Allcott, 2015; Banerjee et al, 2015; Bowers et al, 2017; Davis et al, 2023).…”

Section: Study Motivationmentioning

confidence: 99%

A scientific approach to entrepreneurial decision‐making: Large‐scale replication and extension

Camuffo,

Gambardella,

Messinese

et al. 2024

Strategic Management Journal

View full text Add to dashboard Cite

Research SummaryThis article runs a large‐scale replication of Camuffo and colleagues in 2020, involving 759 firms in four randomized control trials. The larger sample generates novel and more precise insights about the teachability and implications of a scientific approach in entrepreneurship. We observe a positive impact on idea termination and results that are consistent with a nonlinear effect on radical pivots, with treated firms running few over no or repeated pivots. We provide a theoretical interpretation of the empirical results: the scientific approach enhances entrepreneurs' efficiency in searching for viable ideas and raises their methodic doubt because, like scientists, they realize that there may be alternative scenarios from the ones that they theorize.Managerial SummaryThe findings of this article, based on four randomized control trials involving 759 firms, offer new insights into how entrepreneurial practices can benefit from a scientific approach to decision‐making. Key outcomes include an increase in the termination of ideas and a nuanced influence on the tendency to make strategy changes. Specifically, firms that adopted a scientific approach made a few strategic shifts, as opposed to either not changing or constantly changing their strategy. We suggest that this is due to the scientific approach helping entrepreneurs be more efficient when searching for valuable ideas, as well as being more careful in selecting those ideas.

show abstract

Section: Study Motivationmentioning

confidence: 99%

A scientific approach to entrepreneurial decision‐making: Large‐scale replication and extension

Camuffo,

Gambardella,

Messinese

et al. 2024

Strategic Management Journal

View full text Add to dashboard Cite

show abstract

“…The simplicity of sharing materials and procedures is epitomized in the massive collaborations that have been testing the reproducibility of findings in the social sciences at large (Camerer et al 2018), psychology (e.g., Open Science Collaboration 2015Ebersole et al 2016;Klein et al 2018), economics (Camerer et al 2016), and most recently -of landmark experiments published in Management Science (Davis et al 2023). This replicability gave rise to meta-studies: massive experiments run by dozens of researchers, where many interventions are compared for the same outcomes in the same population and duration (Milkman et al 2021).…”

Section: Improve External Validity In Organization Researchmentioning

confidence: 99%

“…Although replication has historically not been highly incentivized (Rubinstein 2001;Ryan and Tipu 2022), the replicability crisis in psychology (Maxwell et al 2015;Shrout and Rodgers 2018) and its spillovers to organizational scholarship (Lewin et al 2016;Pratt et al 2020) have put replication at the top of the research agenda across the social sciences (Lewin et al 2016;Freese and Peterson 2017;Camerer et al 2019;Davis et al 2023), giving a privileged position to experimental methodology and its ability to afford replication and collaboration. The collective response promises to enrich the scientific dialogue and bring us closer to the truth.…”

Section: Improve External Validity In Organization Researchmentioning

confidence: 99%

See 1 more Smart Citation

Primer for Experimental Methods in Organization Theory

Levine,

Schilke,

Kacperczyk

et al. 2023

Organization Science

View full text Add to dashboard Cite

Experiments have long played a crucial role in various scientific disciplines and have been gaining ground in organization theory, where they add unique value by establishing causality and uncovering theoretical mechanisms. This essay provides an overview of the merits and procedures of the experimental methodology, with an emphasis on its application to organization theory. Drawing on the historical roots of experiments and their impact across science, we argue the method holds immense potential for furthering organization theory. We highlight key advantages of experimental methods, including high internal and construct validity, vividness in communicating findings, the capacity to examine complex and understudied phenomena, and the identification of microfoundations and theoretical mechanisms. We alleviate some concerns about external validity and offer guidance for designing and conducting sound, reproducible experimental research. Ultimately, we contend that the current experimental turn holds the potential to reorient organization theory. History: This paper has been accepted for the Organization Science Special Issue on Experiments in Organizational Theory. Funding: This work was supported by the Otto Moensted Foundation, which granted the Otto Moensted Visiting Professorship to S. S. Levine, and a National Science Foundation CAREER Award from the Directorate for Social, Behavioral and Economic Sciences [Grant 1943688] to O. Schilke. Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2023.18093 .

show abstract

“…The Journal of Supply Chain Management (2022) held a virtual publishing workshop in 2022 to introduce registered reports and encourage replication research and regularly reports "Replication Conversations" with operations and supply chain management (OSCM) scholars. Finally, the editorial board of Management Science issued a call for articles to test the replicability of laboratory experiments (cf., Simchi-Levi, 2023), with Davis et al (2023) responding to this challenge by attempting to replicate 10 behavioral operations articles published in Management Science.…”

Section: Introductionmentioning

confidence: 99%

Advancing OSCM scientific knowledge by replicating empirical findings: Step‐by‐step procedure and illustration for transformative replication endeavors

Polyviou,

Rungtusanatham,

Reczek

et al. 2024

Decision Sciences

View full text Add to dashboard Cite

Replication endeavors contribute to the accumulation of scientific evidence about previously reported findings and are crucial for scientific progress. Replication studies are, however, often discouraged and rarely published in the operations and supply chain management (OSCM) discipline. In this article, we offer a framework for replications consisting of two complementary tables. This framework recognizes two types of replications already defined in the literature (i.e., The Exact (EXT) Replication and the Methods‐Only (MTD) Replication) and adds to these two new types (i.e., the Bounded‐Conceptual‐Extension (BCE) Replication and the Transformative (TRF) Replication). The framework clarifies what constitutes replications, forms of replication endeavors, and their purposes. Importantly, we also differentiate replication endeavors from reproducibility tests, robustness checks, and post hoc analyses. Moreover, we describe a seven‐step procedure to guide the design, execution, and presentation of replication endeavors, illustrating these steps by conducting a TRF Replication that incorporates, at the same time, a BCE Replication and an MTD Replication of Polyviou et al. (2018). The proposed framework and seven‐step procedure hopefully motivate OSCM scholars to embrace replications as valuable scientific endeavors that can yield corroborating evidence to bolster confidence in previously reported findings and, better yet, provide new nuanced findings to advance precise scientific understanding of past and new OSCM phenomena.

show abstract

A Replication Study of Operations Management Experiments in Management Science

Cited by 5 publications

References 31 publications

A scientific approach to entrepreneurial decision‐making: Large‐scale replication and extension

A scientific approach to entrepreneurial decision‐making: Large‐scale replication and extension

Primer for Experimental Methods in Organization Theory

Advancing OSCM scientific knowledge by replicating empirical findings: Step‐by‐step procedure and illustration for transformative replication endeavors

Contact Info

Product

Resources

About