Practical Implications of Data Reliability and Treatment Integrity Monitoring

Abstract

Data reliability and treatment integrity have important implications for clinical practice because they can affect clinicians' abilities to accurately judge the efficacy of behavioral interventions. Reliability and integrity data also allow clinicians to provide feedback to caregivers and to adjust interventions as needed. We present reasons why reliability and integrity measures are paramount in clinical work, discuss events that may result in decreased reliability or integrity, and provide several efficient means for collecting data and calculating reliability and integrity measures. Descriptors: Data analysis, integrity, reliability ABSTRACT I

Global Measures of Treatment Integrity May Mask Important Errors in Discrete-Trial Training

“…Peter Pipkin et al 2010;Vollmer et al 2008). The term treatment integrity describes the precision with which interventions are implemented (e.g., Peterson et al 1982).…”

A Microsoft Excel® 2010 Based Tool for Calculating Interobserver Agreement

“…We will, however, reference other literature when warranted. For more information on the practical aspects of IOA data collection (e.g., how often to collect such data, how to interpret and use these data, acceptable levels of IOA), the reader is advised to consult Vollmer et al (2008), also published in Behavior Analysis in Practice.…”

“…As a general rule of thumb, users should only use the most relevant IOA statistic given the nuances of their data collection system and the attributes of the data set, thereby nullifying any need to copy and paste the data across spreadsheet tabs. For assistance in selecting an appropriate IOA statistic, the reader is encouraged to consult Vollmer et al (2008) or Cooper et al (2007).…”

“…Moreover, Vollmer, Sloman, and St. Peter Pipkin (2008) contend that the exclusion of such data substantially limits any interpretation of efficacy of a behavior change procedure. As such, a prerequisite to claims of validity in any study involving behavioral assessment should be the inclusion of reliability data (Friman, 2009).…”

“…Beyond the hurdles of competing responsibilities, Vollmer et al (2008) outline several other barriers to the accurate calculation of reliability. Such barriers include (a) insufficient training on measurement protocols, (b) complexity of the procedure/analysis, and/or, (c) failure to generalize these data collection/analysis skills from instructional situations.…”