On the Performance of the Marginal Homogeneity Test to Detect Rater Drift

Abstract

When constructed response items are administered repeatedly, "trend scoring" can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart's measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to, the paired test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the test had little power. demonstrated good power for all conditions and identified almost all items identified by the test. shows substantial promise for monitoring of trend scoring.

Detecting rater effects in trend scoring

“…For example, a rater's performance may deteriorate over time due to fatigue; that is, they become tired over the course of the scoring project. Other researchers refer to rater drift as changes in rater behavior across test administrations (Park, 2011;Sgammato & Donoghue, 2018). For example, raters might be drawn from a different pool of candidates on every test administration, and on each administration due to multiple factors such as different training personnel, it is not likely that the raters go through exactly the same training as the previous administration.…”

“…This study examines four trend-monitoring statistics: paired t-test and Stuart's (1955) Q for marginal homogeneity, and percent of exact agreement and Cohen's (1960) kappa for interrater agreement. The Q statistic is less well-known than the others being used been found to be more powerful than the t-test (Sgammato & Donoghue, 2018) to detect certain types of changes in rater behavior. The purpose of the present study is to examine the ability of these trend-monitoring statistics to detect rater effects in the context of trend scoring.…”

“…Examples include the many-faceted Rasch model (Linacre, 1989); FACETS model (Lunz, Wright , & Linacre, 1990); an IRT model for multiple raters (Verhelst & Verstralen, 2001); the rater bundle model ; the hierarchical rater model (Patz, Junker, Johnson, & Mariano, 2002) and its signal detection theory version (DeCarlo, 2010;DeCarlo, Kim, and Johnson, 2011); and Yao's rater model (Wang & Yao, 2013). These models are most useful when all the CR items of an assessment have been scored and merged with the multiple choice items (Sgammato & Donoghue, 2018). However in some testing programs, a scoring team consists of a group of 10 to 12 raters who are led by a supervisor and a trainer.…”

“…Within-assessment reliability could be estimated by having some responses scored a second time by a different rater. The most often used statistics to evaluate agreement among raters at one particular time are percentage of exact agreement, correlations, Cohen's (1960) kappa, and the intraclass correlation (Sgammato & Donoghue, 2018).…”

“…In an article that considers the properties of different interrater agreement statistics, Zwick (1988) recommends that assessment of agreement should consist of inspecting marginal homogeneity, specifically with Stuart's Q, and if that holds, then proceed with a measure of agreement. Sgammato and Donoghue (2018) studied two measures related to marginal homogeneity: the paired t-test and Stuart's (1955) Q statistic. They manipulated the sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the score margins.…”