Detecting False Positives in Multielement Designs

Abstract

The authors assessed the extent to which multielement designs produced false positives using continuous duration recording (CDR) and interval recording with 10-s and 1-min interval sizes. Specifically, they created 6,000 graphs with multielement designs that varied in the number of data paths, and the number of data points per data path, using a random number generator. In Experiment 1, the authors visually analyzed the graphs for the occurrence of false positives. Results indicated that graphs depicting only two sessions for each condition (e.g., a control condition plotted with multiple test conditions) produced the highest percentage of false positives for CDR and interval recording with 10-s and 1-min intervals. Conversely, graphs with four or five sessions for each condition produced the lowest percentage of false positives for each method. In Experiment 2, they applied two new rules, which were intended to decrease false positives, to each graph that depicted a false positive in Experiment 1. Results showed that application of new rules decreased false positives to less than 5% for all of the graphs except for those with two data paths and two data points per data path. Implications for brief assessments are discussed.

Detecting False Positives in A-B Designs

“…The results of this study extended findings from the Bartlett et al (2011) study on false positive with multielement designs and provide support for the objectivity of behavior changes detected in A-B designs. In addition, the results of this study potentially strengthen the validity of nonconcurrent multiple baseline designs, which were initially introduced by Watson and Workman (1981) as an experimental design for applied researchers.…”

“…Nevertheless, other researchers have noted that the more immediate the change in the behavior following the introduction of the independent variable, the more likely the change will be attributed to the independent variable (e.g., Horner et al, 2012). Moreover, Bartlett et al (2011) found that the probability of producing false positives in multielement designs decreased with additional sessions. Thus, additional sessions in each phase of an A-B design would likely yield comparable or lower percentages of false positives.…”

“…Figure 3 displays the percentage of the stable 25/random, stable 50/random, and random/random data sets that were scored as either false positives or having the potential to be false positives. The horizontal line denotes the 5% cutoff point, which is a commonly accepted level of false positives in group designs (e.g., Jaccard & Becker, 2002) and was utilized in the Bartlett et al (2011) study. For the stable 25/random data sets, 15.2% were scored as false positives and 27.5% were scored as having potential to be false positives.…”

“…The other form of error, a false negative, occurs when a change in the dependent variable, as a result of the implementation of an independent variable, is not detected when in reality it did occur. Bartlett, Rapp, and Henrickson (2011) argued that false positives could occur as a function of a chosen experimental design, method of data collection, or both. In experimental research, two experimental approaches are commonly used to determine the extent to which a dependent variable changes after the implementation of an independent variable: group designs and single-subject designs.…”

“…In one of the few studies on false positives with single-subject designs, Bartlett et al (2011) generated 6,000 multielement graphs containing two, three, and four data paths (i.e., hypothetical conditions) with two, three, four, or five sessions per data path to represent the different types of multielement graphs often seen in the literature. Bartlett et al found that graphs depicting only two data paths with two sessions per data path produced the highest percentage of false positives, whereas graphs with four or five sessions per data path produced the lowest percentage of false positives.…”