Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data

Abstract

Previous research applying multilevel models to single-case data has made a critical assumption that the level-1 error covariance matrix is constant across all participants. However, the level-1 error covariance matrix may differ across participants and ignoring these differences can have an impact on estimation and inferences. Despite the importance of this issue, the effects of modeling between-case variation in the level-1 error structure had not yet been systematically studied. The purpose of this simulation study was to identify the consequences of modeling and not modeling between-case variation in the level-1 error covariance matrices in single-case studies, using Bayesian estimation. The results of this study found that variance estimation was more sensitive to the method used to model the level-1 error structure than fixed effect estimation, with fixed effects only being impacted in the most extreme heterogeneity conditions. Implications for applied single-case researchers and methodologists are discussed.

Meta-Analysis of Single-Case Experimental Design using Multilevel Modeling

“…Although the REML approach generally performs well with SCED data, it was found that a non-convergence issue could be encountered when analyzing complex multilevel models of SCED data (Baek & Ferron, 2020). Complex models typically involve more parameters, which can be computationally intensive and lead to a non-convergence issue.…”

“…On top of providing a feasible option to solve the non-convergence issue, the Bayesian estimation offers other potential benefits when applied with noninformative and weakly informative priors. Several studies have indicated that the Bayesian estimation can produce precise variance component estimation in a certain condition, more accurate effect size estimation for nonlinear SCED data in comparison to the likelihood estimation (Baek & Ferron, 2020; Baek et al, 2020a; Joo & Ferron, 2019; Moeyaert et al, 2017; Rindskopf, 2014a, 2014b; Shadish et al, 2013; Swaminathan et al, 2014). Therefore, the Bayesian approach can be a practical alternative for researchers who use multilevel models with SCED data, particularly for those estimating a complex model.…”

“…MLM can also account for different types of outcomes in conducting a meta-analysis (Baek et al, 2020b; Moeyaert et al, 2014). Moreover, MLM allows to model heterogeneous variances within participants, across participants, studies, or phases (Baek & Ferron, 2020; Baek et al, 2014; Joo et al, 2019).…”

“…Last, the performance of MLM in analyzing SCED data has been examined using various estimation methods (e.g., Likelihood-based and Bayesian estimations). Several Monte Carlo simulation studies found that treatment effect estimates are unbiased and the inference of the treatment effects are accurate, but variance estimates are typically biased regardless of estimation methods (Baek & Ferron, 2020; Baek et al, 2020a; Moeyaert et al, 2014, 2017; Owens & Ferron, 2012; Petit-Bois et al, 2016; Ugille et al, 2012). Thus, using MLM approach provides a promising result to obtain accurate treatment effects, and provides many potential benefits when meta-analyzing SCED data.…”

Violin plots as visual tools in the meta-analysis of Single-Case Experimental Designs

“…In addition, these techniques are not common ly used in meta-analyses (Jamshidi et al, 2022;Natesan, 2019), probably because they are not well understood by practitioners. Specifically, using multilevel models requires making several complex modeling decisions that may have an impact on the validity of the estimates (Baek & Ferron, 2020;Moeyaert et al, 2016). Moreover, such top-down techniques may not align well with the dominant visual analytical approach, and reduce a large amount of raw data to a single omnibus effect size (Barbosa Mendes et al, 2022) that may not thoroughly capture the complexity of the data (Parker & Vannest, 2012).…”