Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments

The Effects of Incomplete Rating Designs in Combination With Rater Effects

“…For example, methods based on latent trait models such as the many‐facet Rasch (MFR) model (Linacre, ) provide estimates of examinee achievement that are adjusted for systematic differences in rater severity, so long as there are sufficient connections between raters. However, these adjustments require acceptable fit to the MFR model (Wind, Engelhard, & Wesolowski, ). As a result of these statistical adjustments, many researchers and practitioners have used methods based on the MFR model to estimate examinee achievement in rater‐mediated performance assessments (Engelhard & Wind, ).…”

“…Figure includes three examples of popular incomplete rating designs in which there are sufficient links among raters and examinees to facilitate MFR model estimation procedures. These designs appear in both methodological research (e.g., Eckes, ; Engelhard, ; Engelhard & Wind, ; Hombo et al., ; Schumacker, ) and applied research (e.g., Johnson, Penny, & Gordon, ; Wesolowski et al., ; Wind et al., ). In each design, each of the raters scores examinees in common with at least one other rater—thus facilitating the adjustment procedure.…”

Pedagogical Considerations for Examining Rater Variability in Rater‐Mediated Assessments: A Three‐Model Framework

“…Additionally, in some data collection procedures, it may not be feasible for all raters to evaluate all persons used in the study. In rater‐mediated assessments, the connectivity of raters can affect the empirical results of the assessment context (Wind, Engelhard, & Wesolowski, ). Therefore, it is an important research design consideration that warrants researcher specification.…”

“…A data set of example ratings in which 20 raters rated 100 persons on three domains using a five‐category rating scale (0, 1, 2, 3, 4) was simulated to illustrate the interpretation of each component of Equation , where lower numbers indicate lower judged scores of the person and higher numbers indicate higher judged scores of the person. Complete assessment networks, where all raters rate all persons, are theoretically ideal and desirable; however, most large‐scale operational rater‐mediated assessment systems involve various forms of incomplete assessment networks due to time, money, and other administrative constraints (Wind, Engelhard, & Wesolowski, ). As Engelhard () notes, incomplete assessment network designs, when constructed using sound data collection designs, “obtain reliable and valid links both within and between facets that are less costly in terms of examinee time and rater salaries” (p. 27).…”

Exploring the Influence of Range Restrictions on Connectivity in Sparse Assessment Networks: An Illustration and Exploration Within the Context of Classroom Observations

“…Connectivity within teacher evaluation systems is critical because it allows researchers and practitioners to compare teacher performance and principal severity in situations where it is not possible for every principal to rate every teacher. These comparisons cannot be made without systematic connections among different principals and teachers, because it is impossible to separate teachers’ ratings from principals’ severity (Engelhard, ; Linacre, ; Lunz & Linacre, ; Schumacker, ; Wind, Engelhard, & Wesolowski, ). In other words, analyzing teacher evaluation data without taking into account differences between principals can introduce potential bias into the ratings.…”