Forthcoming in Psychological Methods. This paper is not the version of record and may not exactly replicate the final, published version of the article. The final article will be available, upon publication via its DOI. Selective reporting of results based on their statistical significance threatens the validity of meta-analytic findings. A variety of techniques for detecting selective reporting, publication bias, or small-study effects are available and are routinely used in research syntheses. Most such techniques are univariate, in that they assume that each study contributes a single, independent effect size estimate to the meta-analysis. In practice, however, studies often contribute multiple, statistically dependent effect size estimates, such as for multiple measures of a common outcome construct. Many methods are available for meta-analyzing dependent effect sizes, but methods for investigating selective reporting while also handling effect size dependencies require further investigation. Using Monte Carlo simulations, we evaluate three available univariate tests for small-study effects or selective reporting, including the Trim & Fill test, Egger's regression test, and a likelihood ratio test from a three-parameter selection model (3PSM), when dependence is ignored or handled using ad hoc techniques. We also examine two variants of Egger's regression test that incorporate robust variance estimation (RVE) or multi-level meta-analysis (MLMA) to handle dependence. Simulation results demonstrate that ignoring dependence inflates Type I error rates for all univariate tests. Variants of Egger's regression maintain Type I error rates when dependent effect sizes are sampled or handled using RVE or MLMA. The 3PSM likelihood ratio test does not fully control Type I error rates. With the exception of the 3PSM, all methods have limited power to detect selection bias except under strong selection for statistically significant effects.

Publication bias and other forms of outcome reporting bias are critical threats to the validity of findings from research syntheses. A variety of methods have been proposed for detecting selective outcome reporting in a collection of effect size estimates, including several methods based on assessment of asymmetry of funnel plots, such as the Egger's regression test, the rank correlation test, and the Trim‐and‐Fill test. Previous research has demonstated that the Egger's regression test is miscalibrated when applied to log‐odds ratio effect size estimates, because of artifactual correlation between the effect size estimate and its standard error. This study examines similar problems that occur in meta‐analyses of the standardized mean difference, a ubiquitous effect size measure in educational and psychological research. In a simulation study of standardized mean difference effect sizes, we assess the Type I error rates of conventional tests of funnel plot asymmetry, as well as the likelihood ratio test from a three‐parameter selection model. Results demonstrate that the conventional tests have inflated Type I error due to the correlation between the effect size estimate and its standard error, while tests based on either a simple modification to the conventional standard error formula or a variance‐stabilizing transformation both maintain close‐to‐nominal Type I error.

Publication bias and other forms of outcome reporting bias are critical threats to the validity of findings from research syntheses. A variety of methods have been proposed for detecting selective outcome reporting in a collection of effect size estimates, including several methods based on assessment of asymmetry of funnel plots, such as Egger's regression test, the rank correlation test, and the Trim-and-Fill test. Previous research has demonstated that Egger's regression test is mis-calibrated when applied to log-odds ratio effect size estimates, due to artifactual correlation between the effect size estimate and its standard error. This study examines similar problems that occur in meta-analyses of the standardized mean difference, a ubiquitous effect size measure in educational and psychological research. In a simulation study of standardized mean difference effect sizes, we assess the Type I error rates of conventional tests of funnel plot asymmetry, as well as the likelihood ratio test from a three-parameter selection model. Results demonstrate that the conventional tests have inflated Type I error due to correlation between the effect size estimate and its standard error, while tests based on either a simple modification to the conventional standard error formula or a variance-stabilizing transformation both maintain close-to-nominal Type I error.

The purpose of this review was to conduct a meta-analysis of 25 years of mathematics interventions for students with mathematics difficulty or disability in Grades 4 through 12. A search of the literature yielded 25 studies that met the inclusion criteria. Studies were coded to extract important study information (e.g., participant information, research design, description of treatment, and comparison groups) and data needed to calculate Hedge’s g. We used robust variance estimation (RVE) to address dependence resulting from multiple outcomes per study. The RVE random-effects model estimated a treatment effect of 0.85. After adjusting for small-study effects, the final model estimated an underlying, moderate effect of 0.49 with a large amount of unexplained heterogeneity between studies. Studies with more than 15 hr of treatment and those focused on fraction content significantly moderated mathematics outcomes. Findings are limited by extreme variability across study estimates, the lack of standardized mathematics measures, and a limited number of studies across 25 years of research.

Intimate partner violence (IPV) and heavy drinking are co-occurring public health problems, but integrated brief interventions for these conditions have not been tested.OBJECTIVE To determine whether a brief motivational intervention provided at the time of an emergency department (ED) visit reduces IPV and heavy drinking. DESIGN, SETTING, AND PARTICIPANTSA randomized clinical trial conducted at 2 US academic urban EDs between January 2011 and December 2014 to assess the effectiveness of a motivational intervention for IPV-involved female ED patients (ages: 18-64 years; N = 600) who exceeded sex-specific safe drinking limits. All received social service referrals; 2:2:1 to brief intervention (n = 242), assessed control (n = 237), or no-contact control (n = 121).INTERVENTIONS A 20-to 30-minute manual-guided motivational intervention (recorded and monitored for fidelity) delivered by master's-level therapists with a follow-up telephone booster. The assessed control group received the same number of assessments as the brief intervention group; the no-contact control group was assessed only once at 3 months. MAIN OUTCOMES AND MEASURESIncidents of heavy drinking and experiencing IPV measured over prespecified, 12 weekly assessments using an interactive voice response system. RESULTSOf 600 participants, 80% were black women with a mean age of 32 years. Retention was 89% for 2 or more interactive voice response system calls. Seventy-eight percent of women completed the 3-month interview, 79% at 6 months, and 71% at 12 months. During the 12-week period following the brief motivational intervention, there were no significant differences between the intervention group and the assessed control group on weekly assessments for experiencing IPV (odds ratio [OR], 1.02; 95% CI, 0.98-1.06) or heavy drinking (OR, 0.99; 95% CI, 0.96-1.03). From baseline to 12 weeks, the number of women with any IPV in the past week decreased from 57% (134 of 237) in the intervention group to 43% (83 of 194) and from 63% (145 of 231) in the assessed control group to 41% (77 of 187) (absolute difference of 8%). From baseline to 12 weeks, the number of women with past week heavy drinking decreased from 51% (120 of 236) in the intervention group to 43% (83 of 194) and from 46% (107 of 231) in the assessed control group to 41% (77 of 187) (absolute difference of 3%). At 12 months, 43% (71 of 165) of the intervention group and 47% (78 of 165) of the assessed control group reported no IPV during the previous 3 months and 19% (29 of 152) of the intervention group and 24% (37 of 153) of the control group had reduced their alcohol consumption to sex-specific National Institute on Alcohol Abuse and Alcoholism safe drinking levels.CONCLUSIONS AND RELEVANCE For women experiencing IPV and heavy drinking, the use of a brief motivational intervention in the ED compared with assessed and no-contact controls did not significantly reduce the days of heavy drinking or incidents of IPV. These findings do not support a brief motivational intervention in this setting.