Development and Examination of a Tool to Assess Score Report Quality

Abstract

The need for quality in score reporting practices is represented in the Standards for Educational and Psychological Testing (American Educational Research Association American Psychological Association National Council on Measurement in Education, 2014). The purpose of this study was to introduce a ratings-based instrument to assess the quality of score reports and examine the reliability of scores obtained. Quality criteria were derived from best-practices published within the literature (Hambleton and Zenisky, 2013). The rating scale was used to assess a sample of 40 English-language individual student score reports for K-12 accountability tests representing 42 states and five provinces in the United States and Canada. A two-facet generalizability study (i.e., sr x d x r) was completed with an overall reliability coefficient of G = 0.78. Application of the rating scale may provide a means to support empirical study of relationships between score report quality and stakeholder outcomes including interpretation, use, and impact.

State Assessment Score Reporting Practices for English Learner Parents

“…For instance, 100% and 88% of translated academic achievement and ELP reports contained graphics, respectively, which most often came in the form of a score snapshot that clearly provided a summary of results to parents. To increase interpretability of the graphics/scores, many state and ELP score reports contained either a text or graphic description and definition of measurement error (20/37, 54.1%; 42/51, 82.4%, respectively), which has been recommended by prior researchers (e.g., Roduta Roberts & Gotch, 2019).…”

“…Beyond availability of translated materials, several design aspects can be improved upon. One area of need is in better communicating to parents both the purpose of the assessment and how assessment results will be used (see Goodman & Hambleton, 2004; Kim et al., 2016; Roduta Roberts & Gotch, 2019). Most states included the prior, but fewer academic achievement and ELP assessments included statements on intended use.…”

“…To examine score reporting practices, we relied on prior studies to guide our variable coding (e.g., A‐Plus Communications, 1991; Faulkner‐Bond et al., 2013; Goodman & Hambleton, 2004; Hambleton & Zenisky, 2013; Kannan et al., 2018; National Education Goals Panel [NEGP], 1998; Roduta Roberts & Gotch, 2019; Zapata‐Rivera et al., 2014). For each assessment type (state academic achievement and ELP tests), variable coding was conducted separately for score reports and interpretive guides.…”

“…Based on the recommendations of Roduta Roberts and Gotch (2019), we coded for whether score reports contained: (a) a title clearly identifying what the report was for; as well as statements on (b) assessment purpose; (c) intended use of the assessment results; and (d) the knowledge and skills measured by the assessment.…”

“…Based on the research of Roduta Roberts and Gotch (2019), the presence of supplementary information on score reports was investigated. Specifically, we were interested in knowing: (a) whether test results were linked to possible follow‐up activities for improving learning; and (b) the availability of contact information for parents with questions about their child's performance.…”