Interpreting Reports From Universal Screeners: Roadblocks, Solutions, and Implications for Designing Score Reports

Abstract

Many teachers struggle with interpreting formative assessment data. As a result, they are often unable to modify instruction that is responsive to student performance. To support valid interpretations and uses of assessments, test developers need to better understand teachers' ability to read and interpret score reports, their perception of the information that is being conveyed, and how they use the information to change instruction. The purpose of this manuscript is to report on a series of focus groups examining (a) how teachers interact with various score report features and (b) how these features support or hinder their ability to interpret and analyze formative assessment data. Results provide insights into the design of score reports that may facilitate teachers' ability to interpret and analyze data. Solutions to the roadblocks teachers encountered are proposed and generalized examples of how score reports can be designed to facilitate teachers' interpretations are provided. Findings from this study may help test developers anticipate challenges the intended audience could face when interpreting assessment results on score reports, and subsequently improve the quality of score reports to enhance teachers' interpretations and decision-making at the classroom level.

Principals’ and Teachers’ Comprehension of School Performance Feedback Reports. Exploring Misconceptions from a User Validity Perspective

“…However, the literature paints a disconcerting picture with regard to the overall interpretability of score reports (Gotch & French, 2013;Hellrung & Hartig, 2013;O'Leary et al, 2017). On a conceptual level, educational professionals demonstrate a lack of understanding of the constraints of assessment systems (Shivraj & Ketterlin-Geller, 2019) and both criterion-and norm-referenced information in SPF are found to present interpretive challenges (Hellrung & Hartig, 2013). Even basic statistical concepts such as means and percentages have been found to pose problems (Hambleton & Slater, 1997).…”

“…Research exploring disconnects between SPF provider intentions and user interpretations suggests that choice of words and choice of visual presentations matter in score report design. For instance, the amount of specialized and statistical vocabulary to use is a critical consideration (Shivraj & Ketterlin-Geller, 2019) as narrative elements can be too lengthy, too succinct, or otherwise confusing (Hambleton & Slater, 1997). Jargon can be unfamiliar and sometimes intimidating to SPF users, but at the same time vocabulary can also establish tone and authority (Fjørtoft & Lai, 2021;Roduta Roberts et al, 2018).…”

“…Unfortunately, educational professionals often lack the necessary skills and knowledge to effectively interpret data (Hellrung & Hartig, 2013;Hopster-den Otter et al, 2017), as they struggle with comprehending statistical measures and/or visualizations of those measures. In order to determine how SPF can be optimally tailored to educational professionals' data literacy, more insight is needed into actual user interpretations of pupil achievement data (O'Leary et al, 2017;Shivraj & Ketterlin-Geller, 2019;van der Kleij, Eggen, & Engelen, 2014). SPF reports and dashboards are "the primary interface between test developers and […] educational stakeholders" (Gotch & Roduta Roberts, 2018, p. 46) and the way they present information is instrumental in determining whether SPF users will be capable of arriving at valid interpretations.…”