Which Data to Meta-Analyze, and How?

Abstract

Abstract. Which data to analyze, and how, are fundamental questions of all empirical research. As there are always numerous flexibilities in data-analytic decisions (a “garden of forking paths”), this poses perennial problems to all empirical research. Specification-curve analysis and multiverse analysis have recently been proposed as solutions to these issues. Building on the structural analogies between primary data analysis and meta-analysis, we transform and adapt these approaches to the meta-analytic level, in tandem with combinatorial meta-analysis. We explain the rationale of this idea, suggest descriptive and inferential statistical procedures, as well as graphical displays, provide code for meta-analytic practitioners to generate and use these, and present a fully worked real example from digit ratio (2D:4D) research, totaling 1,592 meta-analytic specifications. Specification-curve and multiverse meta-analysis holds promise to resolve conflicting meta-analyses, contested evidence, controversial empirical literatures, and polarized research, and to mitigate the associated detrimental effects of these phenomena on research progress.

(Mis-)understanding COVID-19 and digit ratio: Methodological and statistical issues in Manning and Fink (2020)

“…First, whilst M&F assume that 2D:4D functions as a proxy of prenatal testosterone, it should be noted that scholars have repeatedly pointed to the lack of evidence for this relationship [2][3][4] and 2D:4D findings across different contexts have failed to replicate [5].…”

Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility

“…• Reflects on and verifies the accuracy of the general analysis approach and specific analysis steps taken by original authors. • Applies multiverse and specification-curve approach to the meta-analysis (see Voracek et al 2019). a…”

“…• Deposits beforehand a reproducibility plan or Level 3 checks are performed as outlined for TRANSFER as chosen outcome variable on the (corrected) original dataset; illustrated are own similar and DISsimilar analysis (given there already is a worked example on specification curve for meta-analysis, Voracek et al 2019 • Applies multiverse and specification-curve approach or at least applies own analytical tool and own analytical approaches based on own (re-)creation of (a small portion of) the database by own coding of primary studies.…”

“…Doing so for the multitude of outcome variables reported in some meta-analyses is rather unrealistic. Instead, Voracek et al (2019) suggest a multiverse-analysis approach for meta-analyses identifying theoretically guided eligible specifications that are put into a specification curve. 8 The authors provide a complete worked example and step-by-step guide for meta-analyses.…”

How To Kill A Greek God - A Review, Critique, and Meta-Analysis of 14 years of Proteus Effect Research

“…These decisions may be seen as representing forks in a path that one may choose, with each fork resulting in an alternate outcome (Gelman & Loken, 2013. In approaching this potential source of bias, a form of Multiverse analysis called Combinatorial Meta-Analysis was conducted (see Voracek et al, 2019). In its purest form, Combinatorial MA would involve analysing all possible combinations of studies, which would result in 2 k−1 (i.e.…”