Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature

Hensel, Przemysław

doi:10.1016/j.emj.2021.01.002

Cited by 33 publications

(21 citation statements)

References 117 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other meta-research studies reported similar frequencies of reporting of review protocols (17%) [6], preregistration records (22%) [6], full search strategies for all databases (14%) [7], handling of missing data (25%) [4], or the heterogeneity estimator used (13%) [6]. Some discrepancies in these results can be attributed to differences in assessment criteria and the disciplines studied [38]. In our sample of reviews indexed in 2020, citation of reporting guidelines was common (82%), but was associated with improved reporting of some reporting items only, an observation shared by Wayant et al [4].…”

Section: Discussionmentioning

confidence: 96%

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Nguyen

Kanukula

McKenzie

et al. 2022

Preprint

View full text Add to dashboard Cite

Objectives: To examine changes in completeness of reporting and frequency of sharing review materials in systematic reviews (SRs) over time; and factors associated with these changes. Methods: We examined a random sample of 300 SRs with meta-analysis indexed in PubMed, Science Citation Index, Social Sciences Citation Index, Scopus and Education Collection in November 2020. We compared the extent of complete reporting in these reviews against 110 SRs indexed in MEDLINE in February 2014. We examined associations between completeness of reporting and various factors (e.g. self-reported use of reporting guidelines, journal’s data sharing policies) by calculating risk ratios (RR) and 95% confidence intervals (CI). Results: Among 300 SRs from 2020, authors of only 7% shared their review data file(s) and 1% shared analytic code. Compared to 2014 sample, reviews in 2020 were more likely to reference reporting guidelines in their manuscript (RR=2.8, 95% CI 2.1-3.8), report the full search strategy for at least one database (RR=1.3, 95% CI 1.1-1.6) and methods of data preparation for meta-analysis (RR=2.2, 95% CI 1.4-3.5). Among reviews in 2020, those for which authors mentioned using reporting guidelines reported review protocols, study screening processes and numbers of records retrieved for each database more frequently than those that did not mention reporting guidelines; however the 95% CIs for these associations included the null. Reviews published in journals that mandated either data sharing or inclusion of Data Availability Statements were more likely to share their review materials (e.g. data, code files) (RR=8.1, 95% CI 3.1-21.5). Conclusion: There was a notable increase over time in self-reported use of a reporting guideline, but we were uncertain whether it was associated with improved reporting of SRs. Data sharing policies of journals potentially encourage sharing of review materials. Further studies are needed to explore other facilitators or barriers to complete reporting in SRs.

show abstract

Section: Discussionmentioning

confidence: 96%

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Nguyen

Kanukula

McKenzie

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some researchers have written about a replication and credibility crisis across the social sciences (e.g. Aguinis et al, 2020; Bergh et al, 2017; Hensel, 2021; LeBel et al, 2018). One way to overcome this concern is to include more surveys or samples in each study.…”

Section: Rr Predictorsmentioning

confidence: 99%

Survey response rates: Trends and a validity assessment framework

et al. 2022

View full text Add to dashboard Cite

Survey methodology has been and continues to be a pervasively used data-collection method in social science research. To better understand the state of the science, we first analyze response-rate information reported in 1014 surveys described in 703 articles from 17 journals from 2010 to 2020. Results showed a steady increase in average response rate from 48% in 2005 to 53% in 2010 to 56% in 2015 and 68% in 2020; a marked increase in the number of surveys per published article from 1.27 in 2015 to 1.79 in 2020; and that variables that predict response-rate fluctuations over time are related to research design (e.g. data-collection medium), participant motivation (e.g. incentives), and researcher motivation (i.e. number of surveys per article). Second, we propose complementary information on contemporary response-rate norms and benchmarks with a response-rate validity assessment framework to gather evidence on accuracy of inferences based on a particular response-rate level. Implementing this validation process involves gathering information on the researcher–participant relationship, participant qualifications and motivation, survey length and complexity, and cultural and national context. Future survey research should implement the validity assessment framework in addition to reporting the response-rate value to better indicate a sample’s quality, appropriateness, and representativeness.

show abstract

“…With these findings, our study contributes to an ongoing and very recent discussion about the status quo and value of replication in management research (e.g., Bergh et al, 2017;Dau et al, 2021;De Massis et al, 2020;Hensel, 2021;Köhler and Cortina, 2021;Maula and Stam, 2020;Ryan and Tipu, 2022). By showing an extremely low prevalence of replication studies in the leading management journals, our study adds to the discussion of how open the management field really is to replication studies.…”

Section: Introductionmentioning

confidence: 55%

Replication studies in top management journals: An empirical investigation of prevalence, types, outcomes, and impact

et al. 2022

View full text Add to dashboard Cite

Replication studies are important for the empirical research process. Yet, while there is an increased awareness of the need for replication in management research, it appears that such studies are rarely published in leading management journals. Importantly, we lack a comprehensive overview of replication studies in the top management journals that spans all sub-disciplines. Our systematic review closes this gap and provides an overview of the prevalence, types, outcomes, and impact of replication studies in management journals. We find that differences in the prevalence of replications between sub-disciplines exist and that most replications are wide replications. With regard to the replication outcome, our review shows that the share of non-confirming replications is low. Moreover, such replications are cited less often than confirming replications pointing towards a confirmation bias in management research. We discuss the implications of our results for authors, reviewers, and editors of management journals.

show abstract

Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature

Cited by 33 publications

References 117 publications

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Survey response rates: Trends and a validity assessment framework

Replication studies in top management journals: An empirical investigation of prevalence, types, outcomes, and impact

Contact Info

Product

Resources

About