Agreement between visual inspection and objective analysis methods: A replication and extension

Abstract

Behavior analysts typically rely on visual inspection of single‐case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual‐criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the topic, we examined agreement between the visual inspection of five raters, the conservative dual‐criteria method, and a machine‐learning algorithm (i.e., the support vector classifier) on 198 AB graphs extracted from clinical data. The results indicated that average agreement between the 3 methods was generally consistent. Mean interrater agreement was 84%, whereas raters agreed with the conservative dual‐criteria method and the support vector classifier on 84% and 85% of graphs, respectively. Our results indicate that both objective methods produce results consistent with visual inspection, which may support their future use.

A Comparison of Single-Case Effect Measures Using Check-In Check-Out Data

“…The conservative dual-criterion method (CDC), initially proposed by Fisher et al (2003) for AB designs and later extended to ABAB and multiple baseline designs by Swoboda et al (2010), was applied to the AB contrasts in the data set using the Excel spreadsheets referred to in Swoboda et al (2010). This method has been used to supplement effect measure calculations in another meta-analysis focused on behavioral outcomes (i.e., Maag, 2019), and comparisons between the CDC method and recent research have demonstrated that the CDC method and visual inspection lead to consistent results (Taylor & Lanovaz, 2022;Wolfe et al, 2018). Hence, this method was selected to determine whether systematic change occurred for each AB contrast.…”

“…First, because this manuscript is intended to be a methodological demonstration of how the selection and aggregation of effect measures influence conclusions drawn about the effectiveness of CICO, we elected not to conduct traditional visual analysis and instead used the CDC method. Although recent literature has suggested the consistency between CDC results and visual analysis (Taylor & Lanovaz, 2022; Wolfe et al, 2018), there are differences between these two methods of data analysis that should be acknowledged, especially considering recent recommendations that visual analysis be conducted in addition to calculating effect measures within single-case experimental design research (Moeyaert et al, 2018). Second, only seven effect measures were calculated and analyzed in this review, although many more exist and are used in single-case experimental designs.…”

Making physical activity fun and accessible to adults with intellectual disabilities: A pilot study of a gamification intervention

“…Specifically, the data collected during the baseline and intervention phase for each participants' daily PA, mood, and behaviour identified as challenging by staff were used to create the graphs. The graphs were then interpreted using visual analysus to compare the baseline and intervention data (Lanovaz et al, 2020; Taylor & Lanovaz, 2022). To enhance the reliability of the results, this step was conducted by the first author and an independent researcher who is an experiences behaviour analyst trained in visual analysis of such graphs.…”

Investigation of Two Preliminary Analysis-Altering Elements: Ordinate Scaling and DPPXYR

“…For example, autocorrelation, limited data series, and difficulties with various types of designs make it difficult to identify a standardized solution across experiments. Another possible avenue is using more structured, or objective, methods of visual analysis given the high correspondence with expert visual analysis (Dowdy et al, 2021;Taylor & Lanovaz, 2022); however, structured approaches may threaten, or devalue, clinical expertise in the decision-making process. Limitations to the two aforementioned avenues has led some researchers to continue identifying training methods that can result in improved accuracy and agreement between visual analysts (e.g., Lane et al, 2019).…”