The consequences of modeling autocorrelation when synthesizing single-case studies using a three-level model

Abstract

Results from single-case studies are being synthesized using three-level models in which repeated observations are nested in participants, which in turn are nested in studies. We examined the performance of these models under conditions in which the errors associated with the repeated observations (the Level-1 errors) were assumed to be first-order autoregressive. Monte Carlo methods were used to examine conditions in which the first-order autoregressive assumption was accurate, conditions in which it represented an overspecification because the errors were actually independent, and conditions in which it represented a misspecification because the errors were generated on the basis of a moving-average model. Conditions also varied the series lengths, the numbers of participants per study, the numbers of studies per meta-analysis, the variances between the participants within studies, and the variances between studies. Fixed effects (e.g., the average treatment effect for the intervention and the average treatment effect for the trend) tended to be unbiased, and confidence intervals for the fixed effects tended to be accurate even when the error covariance model was overspecified or misspecified. The variance components, particularly at Levels 2 and 3, showed substantial bias.

Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data

“…The effect of ignoring autocorrelation and other misspecifications of the level-1 error structure have been examined for MLM analyses for single-case data. Researchers found that ignoring autocorrelation does not bias the fixed effect estimates, but the inferences about the fixed effects can be inaccurate due to the underestimate of the corresponding standard errors, and the estimates of the variance parameters become more biased (Ferron et al, 2009;Owens & Ferron, 2012;Petit-Bois et al,2016). Although MLM allows autocorrelation among level-1 errors to be taken into consideration in single-case data analyses, this approach holds a critical assumption that the level-1 error structure is the same for all cases.…”

“…Specifically, it is assumed that (a) autocorrelation is the same for all cases and (b) the level-1 error variance is the same for all cases. Previous single-case research using MLM application as well as misspecification research of level-1 error structures has often assumed the autocorrelation and level-1 error variance to be equal for all cases (e.g., Ferron et al, 2010;Petit-Bois et al, 2016;Van den Noortgate & Onghena, 2003).…”

An examination of measurement procedures and characteristics of baseline outcome data in single-case research

“…Monte Carlo simulations have been used extensively in developing and evaluating methods for analysis of SCD data (e.g., Ferron, Bell, Hess, Rendina-Gobioff, & Hibbard, 2009;Moeyaert, Ugille, Ferron, Beretvas, & Van den Noortgate, 2013;. Simulations are particularly advantageous for studying the robustness of statistical methods because artificial data can be generated based on a model that does not conform to the assumptions of the procedures being studied (e.g., Moeyaert, Ugille, Ferron, Beretvas, & Van den Noortgate, 2016;Petit-Bois, Baek, Van den Noortgate, Beretvas, & Ferron, 2015). Simulations have even been used to examine the performance of analytic methods that are not based on specific distributional assumptions, such as the non-overlap effect sizes for SCDs (e.g., Tarlow, 2017;Pustejovsky, 2018).…”

Meta-Analysis of Single-Case Research via Multilevel Models: Fundamental Concepts and Methodological Considerations

“…Apart from the assumption of independence of residuals from different levels, it is also usually assumed that the Level 1 residuals are independent. For that reason, Petit-Bois, Baek, Van den Noortgate, Beretvas, and Ferron (2016) studied the effect of different specifications of the autoregressive error structure for the measurements. They found that in practically all conditions, the treatment effects are unbiased regardless of whether there is misspecification or not, whereas the estimates of the variance components are biased.…”