Assessing consistency of effects when applying multilevel models to single-case data

Abstract

In the context of single-case experimental designs, replication is crucial. On the one hand, the replication of the basic effect within a study is necessary for demonstrating experimental control.On the other hand, replication across studies is required for establishing the generality of the intervention effect. Moreover, the "replicability crisis" presents a more general context further emphasizing the need for assessing consistency in replications. In the current text, we focus on replication of effects within a study and we specifically discuss the consistency of effects. Our proposal for assessing the consistency of effects refers to one of the promising data analytical techniques: multilevel models, also known as hierarchical linear models or mixed effects models.One option is to check, for each case in a multiple-baseline design, whether the confidence interval for the individual treatment effect excludes zero. This is relevant for assessing whether the effect is replicated as being non-null. However, we consider that it is more relevant and informative to assess, for each case, whether the confidence interval for the random effects includes zero (i.e., whether the fixed effect estimate is a plausible value for each individual effect). This is relevant for assessing whether the effect is consistent in size, with the additional requirement that the fixed effect itself is different from zero. The proposal for assessing consistency is illustrated with real data and it is implemented in free user-friendly software.

Meta-analysis of single-case design research: Application of multilevel modeling.

“…Although the lmer() function in the lme4 R package (Bates et al, 2015) offers a flexible way to handle partially nested data structures, it does not provide an option to model autocorrelations and heterogeneous variances at Level 1, as we did in the current analysis. Manolov and Ferron (2020) noted the advantages of each open software package and cautioned about inconsistencies in the data analysis results. This highlights how software advancements can be related to the development of methodologies and vice versa.…”

Defining and assessing immediacy in single‐case experimental designs

“…Other proposals for assessing consistency have focused on specific designs (see Manolov et al, 2021 , dealing with alternation designs), or specific graphical representations (Manolov & Tanious, 2022 ). Moreover, there have been proposals for assessing consistency of effects in the context of multilevel models (Manolov & Ferron, 2020 ).…”

Defining and assessing immediacy in single-case experimental designs

“…Other proposals for assessing consistency have focused on specific designs (see Manolov et al, 2021, dealing with alternation designs), or specific graphical representations (Manolov & Tanious, 2022). Moreover, there have been proposals for assessing consistency of effects in the context of multilevel models (Manolov & Ferron, 2020).…”