Factors Influencing Intrajudge Consistency During Standard‐Setting

Abstract

What factors influence judges when they set standards? How do judges, test questions, and the standard‐setting process interact? How can we improve intrajudge consistency?

Is an Angoff Standard an Indication of Minimal Competence of Examinees or of Judges?

“…Potential causal factors of inconsistencies both within and between judges have been classified into three categories: (a) the background of judges, (b) the items and their contexts, and (c) the standard-setting processes (Plake et al, 1991). Concerning the judge-related factors Chang et al (1996) showed that when judges differ in their conceptualisation of minimal competency, judgemental inconsistencies may arise.…”

Standard Setting: The Next Generation (Where Few Psychometricians Have Gone Before!)

“…A computerized procedure will also permit constant monitoring of the consistency of the judges' performance Downloaded by [University of Washington Libraries] at 18:28 04 April 2015 during training and facilitate intrajudge consistency during the standard-setting process (Plake et al, 1991). The judges' confidence in their decisions can also be assessed and correlated with their consistency.…”

“…The research on factors that influence judges' ratings during training and the stages of standard setting deal almost exclusively with item judgments and defining minimal competence (Melican & Mills, 1986& Mills, , 1987Mills, Melican, & Ahluwalia, 1991;Norcini, Shea, & Kanya, 1988;Plake, Melican, & Mills, 1991;Pulakos et al, 1989;Smith & Smith, 1988). Despite the limited focus of these studies, the criteria proposed by Reid (1991) for evaluating training effectiveness based on their findings can be generalized to the more recent standard-setting methods: (a) Judgments should be stable over time, (b) judgments should be consistent with item and test score performance, and (c) judgments should reflect realistic expectations.…”

“…The product of this step is a distribution of scored items or work samples at predefined levels with behavioral descriptions and anchors. If this process could be conducted on microcomputers, each judge could be monitored to detect "drift" in the degree of interjudge inconsistency (Plake et al, 1991). 5.…”

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

“…Additionally, iterations aimed at inter-judge consistency are open to "social influence effects by dominant committee members" [37] and "the dynamics of the group discussion", including the desire to appear professional. [7] Consequently, convergence of scores can reflect submission to peer pressure rather than identification of a valid standard.…”