Exploratory Analyses for Missing Data in Meta-Analyses and Meta-Regression: A Tutorial

Abstract

Objectives In this tutorial, we examine methods for exploring missingness in a dataset in ways that can help to identify the sources and extent of missingness, as well as clarify gaps in evidence. Methods Using raw data from a meta-analysis of substance abuse interventions, we demonstrate the use of exploratory missingness analysis (EMA) including techniques for numerical summaries and visual displays of missing data. Results These techniques examine the patterns of missing covariates in meta-analysis data and the relationships among variables with missing data and observed variables including the effect size. The case study shows complex relationships among missingness and other potential covariates in meta-regression, highlighting gaps in the evidence base. Conclusion Meta-analysts could often benefit by employing some form of EMA as they encounter missing data.

On the bias of complete‐ and shifting‐case meta‐regressions with missing covariates

“…Recent empirical work on examining missingness in meta-analytic datasets found that effect sizes can be strongly correlated with missingness, though this is not always the case. 48…”

“…In addition, it is not immediately clear how commonly the conditions required for unbiased complete‐ and shifting‐case estimators arise. Recent empirical work on examining missingness in meta‐analytic datasets found that effect sizes can be strongly correlated with missingness, though this is not always the case 48 . Further, the issues of multicollinearity and confounding in meta‐regression, including those discussed by Lipsey, 25 would suggest that omitting variables in an SCA are likely to induce bias.…”

“…Based on these results, our primary recommendation is that analysts attempt to understand the missingness mechanisms and patterns in their data. This can leverage knowledge about standard reporting and coding practices, as well as exploratory analyses 48 . If there is very little missingness, or if there is a good reason to assume that missingness is uncorrelated with effect size estimates, a CCA may be a reasonable option.…”

“…This can leverage knowledge about standard reporting and coding practices, as well as exploratory analyses. 48 If there is very little missingness, or if there is a good reason to assume that missingness is uncorrelated with effect size estimates, a CCA may be a reasonable option. However, we would discourage analysts from continuing to use SCA because it would seem unlikely that omitted variable biases are zero in practice.…”

Cluster wild bootstrapping to handle dependent effect sizes in meta‐analysis with a small number of studies

“…Applied meta‐analysts are likely to encounter missing data in the moderators because some primary studies may not provide all the information needed to code predictor variables 52 . Schauer and colleagues 53 are currently developing and testing multiple imputation methods for meta‐analyses with dependent data. Once these techniques are more established with typical meta‐analytic methods, future studies should extend them to integrate with cluster wild bootstrapping.…”