Machine Learning to Analyze Single-Case Graphs: A Comparison to Visual Inspection

Abstract

Behavior analysts commonly use visual inspection to analyze single-case graphs, but studies on its reliability have produced mixed results. To examine this issue, we compared the Type I error rate and power of visual inspection with a novel approach, machine learning. Five expert visual raters analyzed 1,024 simulated AB graphs, which differed on number of points per phase, autocorrelation, trend, variability, and effect size. The ratings were compared to those obtained by the conservative dual-criteria method and two models derived from machine learning. On average, visual raters agreed with each other on only 73% of graphs. In contrast, both models derived from machine learning showed the best balance between Type I error rate and power while producing more consistent results across different graph characteristics. The results suggest that machine learning may support researchers and practitioners in making less error when analyzing single-case graphs, but further replications remain necessary.

Agreement between visual inspection and objective analysis methods: A replication and extension

“…We selected the support vector classifier because it produced the fewest errors during the analyses. Furthermore, the support vector classifier agreed more frequently with expert behavior analysts than the behavior analysts amongst themselves in a study with simulated data (Lanovaz & Hranchuk, 2021). The support vector classifier used the eight features extracted from the standardized data (mean, standard deviation, intercept, and slope of each phase) and provided output decisions based on the probability of a clear change in the expected direction.…”

“…For each phase comparison, our code applied a model derived from machine learning to determine whether procedures produced a clear change in the expected direction (see ML_analysis.py for code and ML_Results.py for results of the analysis). Specifically, our analyses involved applying the support vector classifier previously described and developed by Lanovaz and Hranchuk (2021). We selected the support vector classifier because it produced the fewest errors during the analyses.…”

“…The support vector classifier used the eight features extracted from the standardized data (mean, standard deviation, intercept, and slope of each phase) and provided output decisions based on the probability of a clear change in the expected direction. Each of these features represented important characteristics of the data used during visual inspection: Mean is related to level change, standard deviation to variability, intercept to immediacy of change, and slope to trend (Lanovaz & Hranchuk, 2021). Probabilities above, or equal to, 0.5 were categorized as a clear change.…”

“…The support vector classifier can make predictions by examining where a novel data point falls within this higher dimension. In a comparison to visual inspection, Lanovaz and Hranchuk (2021) trained models including the previous algorithm to identify changes in single‐case AB graphs. Their results showed that the support vector classifier produced lower Type I error rates (fewer false positives) and higher power (fewer false negatives) than the conservative dual‐criteria method and visual raters on simulated data.…”

“…We extended Wolfe et al by using a larger number of graphs to examine within‐subject error, blinded normalized graphs, and modified reversal/withdrawal designs. A secondary purpose was to replicate and extend Lanovaz and Hranchuk (2021) by examining correspondence between visual inspection, the conservative dual‐criteria method, and the machine‐learning algorithms on a novel dataset. We extended both studies by using clinical data and adding a 10‐point scale with definitional criteria to visual‐inspection ratings.…”