Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review

Chow, Jason C.; Ekholm, Eric

doi:10.1007/s10648-018-9437-7

Cited by 69 publications

(58 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 3 shows the resulting effect size estimates and associated confidence intervals. There is no clear pattern in terms of higher meta-analytic effect size estimates for published datasets, consistent with previous reports (e.g., Chow & Ekholm, 2018;Guyat et al, 2011), and confidence intervals for the respective sets mostly overlap. Thus, when known factors structuring variance (age,meta-analysis) are accounted for, it does not appear to be the case that the inclusion of grey literature affects meta-analytic conclusions in terms of overall effect size.…”

Section: Meta-analyses With Published and Unpublished Resultssupporting

confidence: 89%

Addressing publication bias in meta-analysis: Empirical findings from community-augmented meta-analyses of infant language development

Tsuji¹,

Cristià²,

Frank³

et al. 2019

Preprint

View full text Add to dashboard Cite

Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two key questions using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent these datasets include grey literature, and by what search strategies they are unearthed. An average of 11% of datapoints are from unpublished literature, and that standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.

show abstract

Section: Meta-analyses With Published and Unpublished Resultssupporting

confidence: 89%

Addressing publication bias in meta-analysis: Empirical findings from community-augmented meta-analyses of infant language development

Tsuji¹,

Cristià²,

Frank³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Figure 3 shows the resulting effect size estimates and associated confidence intervals. There is no clear pattern in terms of higher meta-analytic effect size estimates for published datasets, consistent with previous reports (e.g., Chow & Ekholm, 2018;Guyatt et al, 2011), and confidence intervals for the respective sets mostly overlap. Thus, when known factors structuring variance (age, method, metaanalysis) are accounted for, there does not seem to be a clear pattern as to the direction in which the inclusion of gray literature affects meta-analytic conclusions.…”

Section: Effect Size Estimates By Publication Statussupporting

confidence: 88%

Addressing Publication Bias in Meta-Analysis

Tsuji

Cristià

Frank

et al. 2020

Zeitschrift für Psychologie

View full text Add to dashboard Cite

Abstract. Meta-analyses are an indispensable research synthesis tool for characterizing bodies of literature and advancing theories. One important open question concerns the inclusion of unpublished data into meta-analyses. Finding such studies can be effortful, but their exclusion potentially leads to consequential biases like overestimation of a literature’s mean effect. We address two questions about unpublished data using MetaLab, a collection of community-augmented meta-analyses focused on developmental psychology. First, we assess to what extent MetaLab datasets include gray literature, and by what search strategies they are unearthed. We find that an average of 11% of datapoints are from unpublished literature; standard search strategies like database searches, complemented with individualized approaches like including authors’ own data, contribute the majority of this literature. Second, we analyze the effect of including versus excluding unpublished literature on estimates of effect size and publication bias, and find this decision does not affect outcomes. We discuss lessons learned and implications.

show abstract

“…Various published meta-reviews try to tackle bias by comparing published and unpublished studies (Chow & Ekholm, 2018;Polanin et al, 2016). Since unpublished yet written-up studies may still be subject to various forms of selective reporting of focal results (as most were probably intended to be published), the difference in effect size between published and unpublished results may be a misguiding proxy for the severity of publication bias.…”

Section: Discussionmentioning

confidence: 99%

Neglect of publication bias compromises meta-analyses of educational research

Ropovik¹,

Adamkovič²,

Greger³

2019

Preprint

View full text Add to dashboard Cite

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to a higher prevalence of false positives and the inflation of effect sizes to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last three years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias correction. In the minority of studies adjusting for bias, the outdated trim and fill method was used invariably, and none of the meta-analyses based their conclusions on corrected estimates, rendering the adjustment inconsequential. It is argued that state-of-the-art adjustment (e.g., selection models) should be carried out by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.

show abstract

Do Published Studies Yield Larger Effect Sizes than Unpublished Studies in Education and Special Education? A Meta-review

Cited by 69 publications

References 61 publications

Addressing publication bias in meta-analysis: Empirical findings from community-augmented meta-analyses of infant language development

Addressing publication bias in meta-analysis: Empirical findings from community-augmented meta-analyses of infant language development

Addressing Publication Bias in Meta-Analysis

Neglect of publication bias compromises meta-analyses of educational research

Contact Info

Product

Resources

About