Predictive validity and efficiency of ongoing visual‐inspection criteria for interpreting functional analyses

Abstract

Prior research has evaluated the reliability and validity of structured criteria for visually inspecting functional-analysis (FA) results on a post-hoc basis, after completion of the FA (i.e., post-hoc visual inspection [PHVI]; e.g., Hagopian et al., 1997). However, most behavior analysts inspect FAs using ongoing visual inspection (OVI) as the FA is implemented, and the validity of applying structured criteria during OVI remains unknown. In this investigation, we evaluated the predictive validity and efficiency of applying structured criteria on an ongoing basis by comparing the interim interpretations produced through OVI with (a) the final interpretations produced by PHVI, (b) the authors’ post-hoc interpretations (PHAI) reported in the research studies, and (c) the consensus interpretations of these two post-hoc analyses. Ongoing visual inspection predicted the results of PHVI and the consensus interpretations with a very high degree of accuracy, and PHAI with a reasonably high degree of accuracy. Furthermore, the PHVI and PHAI results involved 32 FA sessions, on average, whereas the OVI required only 19 FA sessions to accurately identify the function(s) of destructive behavior (i.e., a 41% increase in efficiency). We discuss these findings relative to other methods designed to increase the accuracy and efficiency of FAs.

Interpreting functional analysis outcomes using automated nonparametric statistical analysis

“…Functional analysis graphs for inclusion in the present study were identified from the supplemental material file accompanying the study conducted by Saini et al (). In that study, FA datasets had been identified following a thorough review of studies included in the articles by Hanley et al () and Beavers et al () as well as by additional searches using keywords in PsychINFO, ERIC, JABA , and Google Scholar.…”

“…To provide preliminary evidence for the validity of ANSA, we examined the extent to which the ANSA outcomes were concordant with PHAI, the anchor of comparison, as well as PHVI. The outcome for PHAI was determined by corroborating evidence for each dataset from the original published paper and supplementary material (Saini et al, ). An agreement between ANSA and PHAI was scored if ANSA ascribed the same outcome as the published outcome.…”

“…For example, practitioners could run an analysis after each series of FA conditions has been completed to determine if any test conditions are differentiated from the control condition. Practitioners could save time and resources by terminating the assessment as soon as a test condition is identified as being differentiated (Saini et al, ), thereby optimizing accuracy and efficiency of visual analysis methods. One such approach could involve implementing statistical inferential procedures (Fisch, ).…”

“…In the present article, we describe a new statistical approach designed to facilitate FA data interpretation which we term Automated Nonparametric Statistical Analysis (ANSA). Using multielement FA datasets previously identified in the literature by Saini et al (), we first examined the extent to which the data series in each FA satisfied the assumption of normality by testing the distributional characteristics of the conditions in each FA dataset. Given that the majority of data series violated the assumption of normality, we employed distribution‐free or nonparametric statistical procedures rather than parametric procedures.…”

Randomization tests as alternative analysis methods for behavior‐analytic data

“…Despite the utility of skilled visual assessment in determining functions of behavior from functional‐analysis outcomes and the development of structured, formal visual‐analysis protocols for functional analyses (e.g., Hagopian et al, ; Roane, Fisher, Kelley, Mevers, & Bouxsein, ; Saini, Fisher, & Retzlaff, ), assessment of differences in levels of problem behavior between conditions is not always straightforward. Figure , for example, shows aggressions per minute from a hypothetical multielement functional analysis.…”