Evaluating the Impact of Rater Effects on Behavior Rating Scale Score Validity and Utility

Anthony, Christopher J.; Styck, Kara M.; Cooke, Erin A.; Martel, Justin R.; Frye, Katherine E.

doi:10.1080/2372966x.2020.1827681

Cited by 17 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, it is apparent that rater effects do not function monolithically across the motivation/engagement and disruptive constructs examined in this study. This pattern mirrors similar differences across construct domains in other recent work (e.g., Anthony et al, 2022) and is intuitive-raters are individuals that bring their own unique experiences and perspectives on student behavior to the rating task-but problematic. Specifically, rater effects that were consistent within person across domains would likely be easier to identify and remediate.…”

Section: Discussionsupporting

confidence: 83%

“…Indeed, MFRM has only recently been applied to other commonly used school-based behavioral assessments. Specifically, two studies by Anthony et al (2022) and Styck et al (2021) evaluated the presence of rater effects on brief versions of the Social Skills Improvement System-Teacher Rating Scale (Gresham & Elliott, 2008) and the Academic Competence Evaluation Scales (DiPerna & Elliott, 2000). These studies revealed that rater effects were variegated and different in form across different constructs and item formats.…”

Section: Rationale and Purposementioning

confidence: 99%

See 1 more Smart Citation

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Anthony

Styck²,

Volpé³

et al. 2023

School Psychology

Self Cite

View full text Add to dashboard Cite

Although originally conceived of as a marriage of direct behavioral observation and indirect behavior rating scales, recent research has indicated that Direct Behavior Ratings (DBRs) are affected by rater idiosyncrasies (rater effects) similar to other indirect forms of behavioral assessment. Most of this research has been conducted using generalizability theory (GT), yet another approach, many-facet Rasch measurement (MFRM), has recently been utilized to illuminate the previously opaque nature of these rater idiosyncrasies. The purpose of this study was to utilize both approaches (GT and MFRM) to consider rater effects with 126 second-through fifth-grade students who were rated on two DBR-Multi-Item Scales by four raters (22 of these ratings were fully crossed). Results indicated the presence of rater effects and revealed nuances about their nature, including showing differences across construct domains, identifying items that are potentially more susceptible to rater effects than others, and isolating specific raters who appear to have been more susceptible to rater effects than other raters. These findings further indicate the indirect nature of DBRs and offer potential avenues for addressing and ameliorating rater effects in research and practice. Impact and ImplicationsOur study examined whether scores on a popular behavioral assessment, Direct Behavior Ratings-Multi-Item Scales (DBR-MIS), were affected by rater effects, which refer to score differences attributable to rater characteristics rather than student behavior. We used two methodologies that enable a fine-grained examination of these rater effects. Our results illuminated nuanced features of DBR-MIS that hold promise to better address rater effects in future research and practice.

show abstract

Section: Discussionsupporting

confidence: 83%

Section: Rationale and Purposementioning

confidence: 99%

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Anthony

Styck²,

Volpé³

et al. 2023

School Psychology

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several participants expressed concern regarding the influence of rater effects on the data, such as when a particular teacher's ratings appeared to be overly lenient or severe. The presence of rater variance has been confirmed in some prior research examining behavior screening data (Anthony et al, 2021) yet not confirmed in other studies (Lane, Kalberg, Bruhn, et al, 2008). However, it has also been noted that training may help to attenuate these effects (e.g., Splett et al, 2018).…”

Section: Discussionmentioning

confidence: 87%

Exploring Directions for Professional Learning to Enhance Behavior Screening Within a Comprehensive, Integrated, Three-Tiered Model of Prevention

Briesch

Chafouleas

Iovino

et al. 2021

Journal of Positive Behavior Interventions

View full text Add to dashboard Cite

Integration of multitiered academic, behavioral, and social-emotional efforts, such as the implementation of a Comprehensive, Integrated, Three-Tiered (Ci3T) model of prevention, is critical for supporting student development across domains of functioning. In particular, universal behavior screening has been shown to predict outcomes across these domains and as such is foundational to integrated systems. Thus, districts and schools are committed to continuing to learn how to implement behavior screening effectively and efficiently. As such, the purpose of this first qualitative study of systematic screening for behavior was to explore perspectives on benefits, challenges, and opportunities in behavior screening as gathered from members of school- and district-based leadership teams who were currently implementing Ci3T. Both individual interviews and focus groups were conducted, and thematic analysis was used to identify themes that could be used to enhance the implementation and use of behavior screening. Ci3T Leadership Team members indicated knowledge about behavior screening procedures and relayed challenges with respect to using behavior screening data. Taken together, a need emerged for broader professional learning for all school staff members around increasing knowledge and use of behavior screening data. We discuss limitations and directions for future research.

show abstract

“…The second article by Anthony et al (2021) offers a unique evaluation of rater effects on SEB assessments using the Many-Facet Rasch Measurement. Many, if not most, assessments of SEB functioning used for screening, diagnostic, and intervention decisions are rater-mediated assessments (e.g., behavior rating scales; Engelhard & Wind, 2018).…”

Section: Overview Of the Contributing Articlesmentioning

confidence: 99%

“…Although prior research has established the presence and scope of rater effects (e.g., Splett et al, 2018), the typically employed methodology does not allow for evaluation of how rater effects impact score utility and validity. Using a novel methodological approach, Anthony et al (2021) statistically correct rater effects for a sample of students rated on a short form of the Academic Competence Evaluation Scales. Their results indicate few large effects of rater effects on group level score relations (i.e., convergent validity correlations), but more substantial effects at the individual score level.…”

Section: Overview Of the Contributing Articlesmentioning

confidence: 99%

Social, Emotional, and Behavioral Assessment Within Tiered Decision-Making Frameworks: Advancing Research Through Reflections on the Past Decade

Kim

Anthony

Chafouleas³

2021

School Psychology Review

View full text Add to dashboard Cite

Schools play a critical role in service delivery related to students' social, emotional, and behavioral (SEB) health. A foundation for effective SEB service delivery includes capacity to identify students in need of support, differentiate which supports are needed, and monitor students' responsiveness to intervention. Assessment is critical for each of these goals, providing the data needed to make effective decisions across the continuum of service delivery. This special issue provides opportunity to reflect on the past decade of research that strengthens the psychometric evidence for use of the measures in SEB assessment, as well as address several themes that characterize recent SEB assessment research. In this introductory article, we introduce a series of original articles and commentaries addressing these critical issues that are timely and relevant for the future path of SEB assessment. Taken together, the current issue seeks to bring together current evidence around multitiered problem-solving frameworks with focus on SEB domain and propose future directions for research and practice. IMPACT STATEMENTSThe paper describes key themes related to current status and future directions in social, emotional, and behavioral assessment. Those themes are organized around usability, rater effects, a balanced focus on assets and deficits, and equity considerations.

show abstract

Evaluating the Impact of Rater Effects on Behavior Rating Scale Score Validity and Utility

Cited by 17 publications

References 29 publications

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Using many-facet rasch measurement and generalizability theory to explore rater effects for direct behavior rating–multi-item scales.

Exploring Directions for Professional Learning to Enhance Behavior Screening Within a Comprehensive, Integrated, Three-Tiered Model of Prevention

Social, Emotional, and Behavioral Assessment Within Tiered Decision-Making Frameworks: Advancing Research Through Reflections on the Past Decade

Contact Info

Product

Resources

About