Applying Signal‐detection Theory to the Study of Observer Accuracy and Bias in Behavioral Assessment

Abstract

We evaluated the feasibility and utility of a laboratory model for examining observer accuracy within the framework of signal-detection theory (SDT). Sixty-one individuals collected data on aggression while viewing videotaped segments of simulated teacher-child interactions. The purpose of Experiment 1 was to determine if brief feedback and contingencies for scoring accurately would bias responding reliably. Experiment 2 focused on one variable (specificity of the operational definition) that we hypothesized might decrease the likelihood of bias. The effects of social consequences and information about expected behavior change were examined in Experiment 3. Results indicated that feedback and contingencies reliably biased responding and that the clarity of the definition only moderately affected this outcome.

Evaluation of the accuracy, reliability, efficiency, and acceptability of fast forwarding to score problem behavior

“…We attempted to minimize omission error with our instructions by stating, “If you are unsure whether a behavior is problem behavior, we encourage you to score it with a timestamp and behavior code, and then mark it to denote that you want to review that timestamp at normal speed after you have completed viewing the video” (see Supporting Information 4, lines 11‐14). These instructions were similar to Lerman et al’s () manipulation to decrease omissions of aggressive behavior (Experiment 1), which included telling participants, “It's really important to catch all of the aggressions that occur” (p. 201). We were not concerned with observers generating timestamps for nontarget behaviors because they had the option to review those timestamps at normal speed.…”

Data-Based Decision Making: The Impact of Data Variability, Training, and Context

“…Comments collected from our participants suggested that their decisions about the length of baseline would be influenced by information about the type of treatment planned and the form or severity of the targeted behavior. In addition to these variables, studies have suggested that decisions might be influenced by research previously published in the area, the potential significance of the results (DeProspero & Cohen, 1979), or the characteristics of the participants (Kahng et al, 2010; Lerman et al, 2010).…”

Calibration of Observational Measurement of Rate of Responding

“…For applied behavior analysis, exemplar methods for obtaining true reference values against which observers' recordings could be calibrated derive from criterion records. Methods for creating criterion records have included electromechanical recording (e.g., Kapust & Nelson, 1984), using predetermined scripted performances with scripts acting as true records (e.g., Lerman et al, 2010; Powell, Martindale, Kulp, Martindale, & Bauman, 1977), repeated viewing of video records by more than one observer until consensual agreement has been achieved (e.g., Boykin & Nelson, 1981; Mudford, Martin, et al, 2009), and relying on an expert observer who has been trained to high interobserver agreement levels (e.g., Sanson‐Fisher, Poole, & Dunn, 1980; Wolfe, Cone, & Wolfe, 1986).…”