Preliminary Investigation of the Impact of a Web-Based Module on Direct Behavior Rating Accuracy

Abstract

The purpose of this study was to provide initial evaluation of a web-based training module on rating accuracy when using Direct Behavior Rating (DBR). Components of the training module included (a) an overview familiarizing users with assessing student behavior with this method, (b) modeling that includes frame-ofreference training, and (c) multiple opportunities to practice and receive immediate corrective feedback. Participants included 90 undergraduate students assigned to one of six sessions (three experimental and three control). Rating accuracy served as the outcome measure defined as the difference between the rater score and a comparison derived from an expert DBR or systematic direct observation (SDO) score. Rating targets included academically engaged, disruptive, and respectful behavior. Completion of the DBR training module generally yielded ratings that more closely compared with the scores obtained via DBR experts and SDO, yet specific results were mixed across type of rating (i.e., behavior target and duration) and comparison (i.e., DBR expert and SDO). Limitations, future research directions, and implications for practice are discussed.

Brief Online Training with Standardised Vignettes Reduces Inflated Supervisor Ratings of Trainee Practitioner Competencies

“…In the context of a brief online training task, these vignettes familiarise supervisors with the performance and competency standards expected at particular points along the developmental continuum. This is consistent with other research demonstrating the effectiveness of training in improving rater reliability and accuracy across varied disciplines (Chafouleas et al, 2015;Jelley & Goffin, 2001;Schanche et al, 2010;Stamoulis & Hauenstein, 1993;Støre-Valen et al, 2015;Thornton & Zorich, 1980). It may be that providing clear frames of reference indirectly attenuates vulnerability to rater biases that is commonly observed in CERFs (Dudek et al, 2005;Gonsalvez et al, 2015;Stamoulis & Hauenstein, 1993).…”

“…Therefore, it is important that supervisors have access to, and engage with, resources that are designed to support this role. Given the high degree of responsibility and time constraints placed on field supervisors, these resources need to be easily accessible, time efficient, effective, and applicable to a wide variety of placement types and settings (Chafouleas, Riley‐Tillman, Jaffery, Miller, & Harrison, 2015; Olsen, Donaldson, & Hudson, 2010). Given that field supervisors come from diverse settings, any resources designed to assist in the accurate assessment of trainees should be standardised to ensure that there is consistency in supervisors’ expectations of trainees and in the assessment process.…”

“…Prior research in other disciplines and settings (e.g., medicine, clinical assessment, behavioural assessment of children, occupational skills assessment) has suggested that behavioural assessment training (both online and in vivo) is a means for reducing bias and improving rating accuracy (Chafouleas et al, 2015; Jelley & Goffin, 2001; Schanche, Høstmark Nielsen, McCullough, Valen, & Mykletun, 2010; Stamoulis & Hauenstein, 1993; Støre‐Valen et al, 2015; Thornton & Zorich, 1980). For instance, online training has been shown to improve the reliability of Global Assessment of Functioning (GAF)1 scores (Støre‐Valen et al, 2015).…”

The Effects of CellF-Monitoring on Students’ Academic Engagement: A Technology-Based Self-Monitoring Intervention

“…Students began using the application and compared responses to their teacher's ratings of AE at the end of each day's designated independent work time. The training phase continued until three consecutive days of teacher and student ratings that fell within 20% of one another, based on criteria set by Chafouleas et al (2015).…”

“…Three of the six eligible third- through fifth-grade teachers expressed interest in participating, signed consent forms, and completed the student nomination interview. The teachers completed the 45-min long online training module for the DBR-SIS, which consisted of an overview of the measure, a model and frame-of-reference training, and multiple opportunities to practice and receive immediate corrective feedback (Chafouleas, Riley-Tillman, Jaffery, Miller, & Harrison, 2015). Subsequently, teachers rated the nominated student’s AE to ensure ratings below a seven.…”