An evaluation of the agreement between the conservative dual‐criterion method and expert visual analysis

Wolfe, Katie; Seaman, Michael A.; Drasgow, Erik; Sherlock, Phillip

doi:10.1002/jaba.453

Cited by 35 publications

(41 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results also extend prior research on the use of the DC method for the analysis of single-case data. Agreement between the expert rater and the DC method was .869, which is a value consistent with recent research on the topic (Wolfe et al, 2018). As more research is conducted, practitioners and researchers may eventually use machine learning to supplement visual analysis.…”

Section: Discussionsupporting

confidence: 85%

“…Moreover, Lanovaz, Huxley, and Dufour (2017) reported that the DC method provided adequate control over Type I error rate with nonsimulated data. In a recent study, Wolfe, Seaman, Drasgow, and Sherlock (2018) evaluated the rate of agreement between the CDC method and expert visual analysts on 66 AB tiers from published multiple baseline graphs. Their findings show that mean agreement between the CDC method and expert visual analysts was 84%, indicating that the CDC method may still yield incorrect classifications in one of six cases.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning to Analyze Single-Case Data: A Proof of Concept

2020

View full text Add to dashboard Cite

Visual analysis is the most commonly used method for interpreting data from singlecase designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach.

show abstract

Section: Discussionsupporting

confidence: 85%

mentioning

confidence: 99%

Machine Learning to Analyze Single-Case Data: A Proof of Concept

2020

View full text Add to dashboard Cite

show abstract

“…For single-case research, decision on the effect of the independent variable primarily relies on visual inspection of graphs, which is also recommended by the WWC guidelines (Kratochwill et al, 2010(Kratochwill et al, , 2013. Although visual inspection remains a subjective practice (Manolov & Vannest, 2020;Ninci et al, 2015;Wolfe et al, 2018), the Type I error rate and the power of visual inspection are largely undocumented. As such, one may argue that visual inspection could be more powerful than the dual-criteria method and circumvent the issues observed in the first study.…”

Section: Study 2 -Error Rate and Power With Visual Inspectionmentioning

confidence: 99%

How Many Tiers Do We Need? Type I Errors and Power in Multiple Baseline Designs

Lanovaz¹,

Turgeon²

2020

Preprint

View full text Add to dashboard Cite

Design quality guidelines typically recommend that multiple baseline designs include at least three demonstrations of effects. Despite its widespread adoption, this recommendation does not appear grounded in empirical evidence. The main purpose of our study was to address this issue by assessing Type I error rate and power in multiple baseline designs. First, we generated 10,000 multiple baseline graphs, applied the dual-criteria method to each tier, and computed Type I error rate and power for different number of tiers showing a clear change. Second, two raters categorized the tiers for 300 multiple baseline graphs to replicate our analyses using visual inspection. When multiple baseline designs had at least three tiers and two or more of these tiers showed a clear change, the Type I error rate remained adequate (< .05) while power also reached acceptable levels (> .80). In contrast, requiring all tiers to show a clear change resulted in overly stringent conclusions (i.e., unacceptably low power). Therefore, researchers and practitioners should carefully consider limitations in power when requiring all tiers of a multiple baseline design to show a clear change in their analyses.

show abstract

“…Researchers have sought to counter and directly address these criticisms by developing standardized protocols for assessing the credibility of single-case research design and evidence, 12,20,21 data analysis, reporting guidelines for publishing single-case research findings, 22 and the use of standardized visual analysis practices to determine the strength of the functional relation 23,24 and have validated training packages that can be used to train students and scholars in visual analysis. 14,25,26 For instance, the United States Department of Education Institute for Education Science (IES) White Paper defines the What Works Clearinghouse Pilot Standards for single-case design and analysis.…”

Section: Special Issue On Advances In Single-case Research Design Andmentioning

confidence: 99%