Application of Latent Trait Models to Identifying Substantively Interesting Raters

Abstract

Historically, research focusing on rater characteristics and rating contexts that enable the assignment of accurate ratings and research focusing on statistical indicators of accurate ratings has been conducted by separate communities of researchers. This study demonstrates how existing latent trait modeling procedures can identify groups of raters who may be of substantive interest to those studying the experiential, cognitive, and contextual aspects of ratings. We employ two data sources in our demonstration—simulated data and data from a large‐scale state‐wide writing assessment. We apply latent trait models to these data to identify examples of rater leniency, centrality, inaccuracy, and differential dimensionality; and we investigate the association between rater training procedures and the manifestation of rater effects in the real data.

Features of difficult-to-score essays

“…Previously, Wolfe and McVay (2012) criticized much of the extant research regarding rater effects because it failed to jointly consider quantitatively-precise definitions and measures of rating quality with substantive variables that have been thoughtfully selected based on detailed models of the content (in our case, writing). That is, our study demonstrates one way that traditional psychometric methods can be coupled with automated scoring technologies and the associated deep substantive knowledge base upon which those technologies are based to produce results that are important from both quantitative and substantive perspectives.…”

Rater Cognition: Implications for Validity

“…In short, prior to the act of scoring itself, much preparation takes place, including the training of the raters that ideally leads each rater to a common mental rubric. In practice, a variety of factor can affect the mental rubric such that the scoring behavior of raters is far from identical and can lead to a number of rater effects Such rater effects are the overt manifestation of lack of exchangeability among raters and can be detected with the appropriate psychometric tools (see Wolfe, 2012).…”

Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments

“…For example, assessment instruments that are better aligned with content as well as the target context have been shown to increase reliability; however, these do not appear to adequately address examiner variation (Crossley et al 2011). Examiner training has also been explored as a means of supporting examiners and increasing reliability (e.g., Green and Holmboe 2010;Wolfe and McVay 2012), but the effectiveness of such training has been inconsistent. Finally, frame of reference training has also been suggested as a means of increasing the reliability of scores and the validity of inferences with performance assessments (Kogan et al 2011), though there remain debates as to what constitutes an appropriate frame of reference.…”