Using Visual Analysis to Evaluate and Refine Multilevel Models of Single-Case Studies

Abstract

In special education, multilevel models of single-case research have been used as a method of estimating treatment effects over time and across individuals. Although multilevel models can accurately summarize the effect, it is known that if the model is misspecified, inferences about the effects can be biased. Concern with the potential for model misspecification motivates our method for evaluating multilevel models of single-case data. This method is based on the visual analysis of graphs that have the model-implied individual trajectories superimposed on plots of the raw data. Through the reanalysis of a published study, we show how this visual analysis approach can identify model misspecifications and motivate the consideration of alternative model specifications that lead to better fit.

Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data

“…For example, discovered relatively large differences in estimates of autocorrelation and level-1 error variances, when they allowed the level-1 error structure to vary across cases in real datasets from four previously published single-case studies (i.e., Dufrene et al, 2010;Ingersoll & Lalonde, 2010;Koegel, Singh, & Koegel, 2010;Oddo et al, 2010), and the fit indices favored a model with separate estimates of the autocorrelation and the level-1 error variances for some studies. Baek et al (2016) also found that allowing the level-1 error structure to differ for one of the participants in a single-case study led to estimated individual trajectories that were more consistent with the visually plotted data and improved the model fit.…”

“…The level-1 error SD were generated such that the largest level-1 error variance value can be either as much as 3.5 times or as much as 16 times the smallest level-1 error variance value. The motivation was based on the analyses of real single-case design datasets presented by and Baek et al (2016). The level-2 errors u 0j , u 1j , u 2j , and u 3j , were generated independently from normal distributions with mean 0 and variances of either 0.5, 0.5, 0.05, and 0.05, or 2, 2, 0.2, and 0.2, respectively.…”

MultiSCED: A tool for (meta-)analyzing single-case experimental data with multilevel modeling

“…7) are far from the observed data, this might indicate a model misspecification. In Baek et al (2016) for example, a similar two-level model including a trend is fit. Upon inspection of the individual Bayes trajectories (p. 21, Figure 2), the authors state that 'because, overall, the visual result of the first model appeared to be poor,' they are motivated to 'consider an alternative model that may represent the data better.…”

Extrapolating baseline trend in single-case data: Problems and tentative solutions

“…In that sense, representing visually the trend line fitted and extrapolated or the transformed data after baseline trend has been removed is crucial. Accordingly, recent efforts have focused on using visual analysis to help choose the appropriate multilevel model (Baek, Petit-Bois, Van Den Noortgate, Beretvas, & Ferron, 2016). To make more transparent what exactly is being done with the data to obtain the quantifications, the output of the modified MPD is both graphical and numerical (see http://manolov.shinyapps.io/ MPDExtrapolation, which allows for choosing whether to limit the extrapolation of the baseline trend and whether to use damping or winsorizing in the case of out-of-bounds forecasts).…”