Interobserver agreement and procedural fidelity: An odd asymmetry

Abstract

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior‐analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission.

A detailed examination of reporting procedural fidelity in the Journal of Applied Behavior Analysis

“…Furthermore, authors may also report interobserver agreement between two observers assessing procedural fidelity as the fidelity measure (e.g., Dyer et al, 1982), but agreement, by itself, is not an indication of procedural fidelity (Kazdin, 1977). A limitation of the current review is that there was no assessment of IOA of procedural‐fidelity data, but Essig et al (2023) reported that 17% of articles published in JABA between 2017 and 2021 reported interobserver agreement for procedural fidelity. Currently, there is no minimum standard for the proportion of sessions that include interobserver agreement for procedural fidelity, but the standard should be at least as rigorous as those for the dependent variable (i.e., 20% of data points within each phase; Collier‐Meek et al, 2018; What Works Clearinghouse, 2022).…”

“…In addition to using fidelity and integrity interchangeably, authors have referred to fidelity in multiple ways including (e.g., “adherence to training” in Jones et al, 1981; “reliability” in Ollendick et al, 1983; and “independent variable accuracy” in Bachman & Fuqua, 1983). The current review focused exclusively on contemporary literature for which the terms fidelity and integrity were more likely to be used (see Essig et al, 2023); experiments that assessed fidelity but used a different term were excluded. Previous studies have noted the variety of terms used to describe fidelity as a barrier to measurement (e.g., Hagermoser Sanetti & DiGennaro Reed, 2012).…”

“…Therefore, these data may not reflect reporting practices in other behavior-analytic journals and additional research should be conducted. Our hope is that the current data will serve as an inflection point (Essig et al, 2023) against which the progress of further evaluations can be compared.…”

On a persisting curious double standard in behavior analysis: Behavioral scholars' perspectives on procedural fidelity

“…Scholars of all levels noted receiving infrequent feedback through the editorial process about the necessity of fidelity data, despite the impression that parallel data for the dependent variable (i.e., IOA) would be required (see Essig et al, 2023, for a recent commentary on the relation between IOA and fidelity data). For example, one mid-career scholar noted, "It's the contingencies in place that will reinforce my behavior of collecting IOA data right now, more regularly than trying to juggle and have a decent integrity measure at the same time.…”

Comparing a modified simple‐conditional with the conditional‐only methods in teaching Chinese children with autism

“…Fourth, we did not examine interobserver agreement of the procedural‐fidelity data. Future studies should consider assessing the interobserver agreement of their procedural fidelity data (Essig et al, 2023). Finally, all participants had a history of receiving the conditional‐only method during their regular intervention sessions.…”