A Framework for Evaluating Reproducibility and Replicability in Economics

Dreber, Anna; Johanneson, Magnus

doi:10.2139/ssrn.4458153

Cited by 7 publications

(3 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If not indicated otherwise in Supplementary Methods, Sections 1-3, we were able to precisely (computationally) reproduce the results reported in the papers. As such, our study provides-as a "side product"-evidence on the computational reproducibility [97][98][99][100][101] of large-scale meta-scienti c results.…”

Section: Estimation Of Results For Included Studiesmentioning

confidence: 88%

Heterogeneity in effect size estimates: Empirical evidence and practical implications

Holzmeister,

Johannesson,

Böhm

et al. 2024

Preprint

View full text Add to dashboard Cite

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confidence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confidence intervals. We illustrate that heterogeneity of this magnitude—unless properly accounted for—has severe implications for statistical inference with strongly increased rates of false scientific claims.

show abstract

Section: Estimation Of Results For Included Studiesmentioning

confidence: 88%

Heterogeneity in effect size estimates: Empirical evidence and practical implications

Holzmeister,

Johannesson,

Böhm

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Several definitions of reproducibility and replicability have been used and proposed (see, e.g., Clemens 2017;Christensen and Miguel 2018;Ankel-Peters et al, 2023a). Dreber and Johannesson (2023) recently proposed definitions and indicators for economics that we summarize here and believe can be useful also for political science. As mentioned above, reproducibility is defined as using the original studies' data, while replicability is defined as using data other than what was used in the original studies.…”

Section: Definitions Of Reproducibility and Replicability In Politica...mentioning

confidence: 99%

“…Conceptual replicability also uses new data to test to what extent results in original studies can be repeated, but for this type of replication an alternative research design and/or alternative analysis is used to test the same hypothesis as in the original study (and conceptual replicability is also further subdivided into the same three sub-categories as direct replicability). See Dreber and Johannesson (2023) for further details and proposed indicators of reproducibility and replicability to be reported for each type.…”

Section: Definitions Of Reproducibility and Replicability In Politica...mentioning

confidence: 99%

Promoting Reproducibility and Replicability in Political Science

Brodeur,

Esterling,

Ankel-Peters

et al. 2024

Research & Politics

Self Cite

View full text Add to dashboard Cite

This article reviews and summarizes current reproduction and replication practices in political science. We first provide definitions for reproducibility and replicability. We then review data availability policies for 28 leading political science journals and present the results from a survey of editors about their willingness to publish comments and replications. We discuss new initiatives that seek to promote and generate high-quality reproductions and replications. Finally, we make the case for standards and practices that may help increase data availability, reproducibility, and replicability in political science.

show abstract

Experiments in finance

Kirchler

Weitzel

2023

Journal of Banking & Finance

View full text Add to dashboard Cite

A Framework for Evaluating Reproducibility and Replicability in Economics

Cited by 7 publications

References 73 publications

Heterogeneity in effect size estimates: Empirical evidence and practical implications

Heterogeneity in effect size estimates: Empirical evidence and practical implications

Promoting Reproducibility and Replicability in Political Science

Experiments in finance

Contact Info

Product

Resources

About