Revealed or Concealed? Transparency of Procedures, Decisions, and Judgment Calls in Meta-Analyses

Abstract

The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.Keywords meta-analysis, quantitative research, missing dataAs the amount of research in a field increases, the need to integrate the findings increases in tandem. Over the past 30 years, systematic review and meta-analysis have moved from being somewhat controversial to generally being a preferred way of integrating research findings in many scientific disciplines (Cooper, 2009;Hunt, 1997;Wanous, Sullivan, & Malinak, 1989). Evidence of the growing dependence on meta-analysis to summarize research literatures comes in at least two forms: the increase in the number of meta-analyses published and the increase in the number of citations of meta-analyses over time. Guzzo, Jackson, and Katzell (1987) charted journal articles and dissertations appearing in Psychological Abstracts between 1977 and 1985 that used or commented on,

Publication Bias in the Organizational Sciences

“…Aytug, Rothstein, Zhou, and Kern (2012) provided a relatively comprehensive account on the information that should be included in a meta-analytic review to ensure transparency and replicability not only of the search, but the meta-analytic review in general. This information could be submitted as supplementary material and made available online by the journal.…”

Meta-Synthesis of Qualitative Case Studies

“…This codification allows the readers to retrace the certain paths of the meta-synthesis process as well as to critically evaluate the process and its associated product. For example, by providing a meta-synthesis protocol, transparently reporting the literature search and the inclusion/exclusion criteria as well as indicating the procedures for coding and analysis, not only the validity of the results, but also the reliability of the meta-synthesis study itself can be ensured; an aspect that is generally considered as one of the key advantages of research synthesis over literature reviews Aytug et al, 2012;Dalton & Dalton, 2008).…”

“…As a critical decision in research synthesis entails what features to code (Aytug et al, 2012;Kisamore & Brannick, 2008), the process of extracting and coding data was guided by a reliable and valid coding form. To ensure inter-reviewer consistency during the course of coding, any discrepancies that emerged were carefully documented in the coding form and resolved by discussions and further rereadings of the original studies.…”

“…Furthermore, the use of complementary electronic and manual search techniques ensured that material is not missing either through the inadequacies of indexing or through the selective coverage of databases. Taken together, a systematic, explicit, and transparent search process generates a rigorous meta-synthesis study, thereby acknowledging that ill-defined or biased searches are likely to result in an inadequate database and later, inaccurate results (Aytug et al, 2012;Cooper, 2010).…”

Guest Editor’s Introduction to Special Issue: The Science and Practice of Research Synthesis

“…Recent studies point to the need for improvements in the conduct and reporting of published systematic reviews and meta-analyses (Aytug, Rothstein, Zhou, & Kern, 2012;Moher, Tetzlaff, Tricco, Sampson, & Altman, 2007). Because the growth in scholarly production has not yet begun to plateau, we will need leaner, more efficient systematic review methods to help decision makers keep up with the evidence (Bastian et al, 2010).…”