Using Calibration and Interobserver Agreement Algorithms to Assess the Accuracy and Precision of Data From Electronic and Pen‐and‐paper Continuous Recording Methods

Abstract

Often it is assumed that electronic recording by observers necessarily provides better quality data than pen-and-paper methods. Fifteen novice observers recorded rates of responding from 10 role-played video samples using one of three continuous recording input formats: keyboard (laptop), touchscreen (personal digital assistants), or pen-and-paper. We evaluated the quality of the observers' data compared with criterion records using calibration and interobserver agreement algorithms. Results of the calibration analysis revealed that observers in the touchscreen group produced the most consistently accurate and precise data, the keyboard group observers showed wide variation in precision and accuracy, and the pen-and-paper group observers were significantly less precise than the touchscreen group. We conclude that although electronic recording has the potential to be as accurate as, and more precise than, pen-and-paper methods, this is far from guaranteed. Analyses of observers' errors advise recommendations for improving data accuracy and precision when using each method.

Accuracy and acceptability of eHealth data collection for an early intensive behavioral intervention program

“…Our findings differed from those of Phillips et al. (2014), where the touchscreen condition was more accurate and precise than pen‐and‐paper for one participant. This discrepancy may be a function of the observation systems in each study.…”

“…Electronic data collection (EDC) has the potential to address many limitations of pen‐and‐paper recording (Phillips et al., 2014). Copies of data may be produced instantly and with limited environmental impact.…”

“…The EDC tool required more time for data collection but less time for graphing session data. Phillips et al (2014) compared the accuracy and precision of data produced by pen-and-paper with two different EDC systems (a laptop keyboard and a touchscreen personal digital assistant). Participants recorded the frequency of a challenging behavior ("client pinching therapist") in scripted videos of confederates role-playing teaching sessions.…”

Different Methods for Long-term Systematic Assessment of Challenging Behaviors in People with Severe Intellectual Disability

“…Phillips et al (2014) conducted a study with 15 observers recording response rates from 10 video clips using one out of three input formats: keyboard (laptop), touch screen (digital assistants personal), or paper-and-pencil. The results showed that although the electronic record had the potential to be more accurate and more reliable than paper-and-pencil method, this was far from being conclusive.…”

Computer‐based training to teach observers to accurately score problem behavior using fast forwarding at 5x normal speed

“…Internet‐connected cameras allow professionals to obtain extended recordings of caregiver–client interactions in natural settings, and hand‐held computers, tablets, and smartphones have made continuous measurement easier to conduct (Phillips et al, 2014). However, it is impractical to score extended observations at normal speed to measure instances of low‐rate behavior.…”