Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews

Abstract

BackgroundSystematic reviews (SRs) are an important source of information about healthcare interventions. A key component of a well-conducted SR is a comprehensive literature search. There is limited evidence on the contribution of non-English reports, unpublished studies, and dissertations and their impact on results of meta-analyses.MethodsOur sample included SRs from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of reviews that searched for and included each study type; 2) proportion of relevant studies represented by each study type; and 3) impact on results and conclusions of the primary meta-analysis for each study type.ResultsMost SRs searched for non-English studies; however, these were included in only 12% of reviews and represented less than 5% of included studies. There was a change in results in only four reviews (total sample = 129); in two cases the change did not have an impact on the statistical or clinical significance of results. Most SRs searched for unpublished studies but the majority did not include these (only 6%) and they represented 2% of included studies. In most cases the impact of including unpublished studies was small; a substantial impact was observed in one case that relied solely on unpublished data. Few reviews in ARI (9%) and ID (3%) searched for dissertations compared to 65% in DPLP. Overall, dissertations were included in only nine SRs and represented less than 2% of included studies. In the majority of cases the change in results was negligible or small; in the case where a large change was noted, the estimate was more conservative without dissertations.ConclusionsThe majority of SRs searched for non-English and unpublished studies; however, these represented a small proportion of included studies and rarely impacted the results and conclusions of the review. Inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are questionable vested interests in the published literature. We found substantial variation in whether SRs searched for dissertations; in most reviews that included dissertations, these had little impact on results.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-017-0347-z) contains supplementary material, which is available to authorized users.

Is bullying and teasing associated with eating disorders? A systematic review and meta‐analysis

“…Last, we did not include grey literature in our search. However, it is doubtful that inclusion of such literature would alter our main findings, and one recent study shows that inclusion of such literature has limited impact on reviews (Hartling et al, ).…”

How should systematic reviewers handle conference abstracts? A view from the trenches

“…Second, largely driven by their brevity, abstracts may not contain adequate information for systematic reviewers to appraise the design, methods, risk of bias, outcomes, and results of studies reported in the abstracts [18][19][20][21]. Third, the dependability of results presented in abstracts also is questionable [22][23][24], which occurs at least in part because (1) most abstracts are not peer-reviewed and (2) results reported in abstracts are often preliminary and/or based on limited analyses conducted in a rush to meet conference deadlines. The most frequent types of conflicting information between abstract and full-length journal article have pertained to authors or authorship order, sample size, and estimates of treatment effects (their magnitude or, less frequently, direction) [25][26][27][28][29][30][31].…”

Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review

“…Searches were conducted from the inception of each database to May 2017 (Table S1). In line with guidance relating to searching, additional methods were adopted to improve the comprehensiveness of the searches (Brettle and Grant, 2004;Stansfield et al, 2016;Hopewell et al, 2007;Hartling et al, 2017…”