Comparing Construct Definition in the Angoff and Objective Standard Setting Models

Abstract

Typical validation studies on standard setting models, most notably the Angoff and modified Angoff models, have ignored construct development, a critical aspect associated with all conceptualizations of measurement processes. Stone compared the Angoff and objective standard setting (OSS) models and found that Angoff failed to define a legitimate and stable construct. The present study replicates and expands this work by presenting results from a 5-year investigation of both models, using two different approaches (equating and annual standard setting) within two testing settings (health care and education). The results support the original conclusion that although the OSS model demonstrates effective construct development, the Angoff approach appears random and lacking in clarity. Implications for creating meaningful and valid standards are discussed.

The Objective Borderline Method: a probabilistic method for standard setting

“…Most studies compare the results of one standard setting method to another, either by applying different methods on the same scores or by comparing decisions made at one examination with future assessment outcomes (Buckendahl et al 2002;Wayne et al 2005;George, Haque, and Oyebode 2006;Wood, Humphrey-Murto, and Norman 2006;Schoonheim-Klein et al 2009;Shulruf et al 2013;Stone, Koskey, and Sondergeld 2011). Although such comparisons are probably the best practice available when using real data, the students' true abilities remain unknown.…”

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

“…This is despite the considerable effort made at the start of the standard setting process to allow judges to construct their own conceptual notion of MCE in relation to safe clinical practise. [40] Moreover, if judges are drawn towards the concept of average student when trying to conceptualize a MCE, the above standard setting process shares a close affinity with normreferenced approaches, [41] which are typically regarded as inappropriate for high stakes assessments in medical education. [19] Clearly, iterative approaches to standard setting also incur considerable financial and administrative burdens in terms of training judges and employing them over several days, including reimbursement of health authorities for releasing employees.…”

“…OSS, as presented in this paper, refers specifically to the case of itemized written exams. [40] Given a pool of potentially selectable items (PSIs) representative of a given field, judges must decide between two content-based options for each item. For the given item, the content is either i) essential for a candidate of the chosen competency level to have mastered, where the interpretation of 'essential' has first been discussed and agreed on, or ii) "important, but not essential".…”

“…For the given item, the content is either i) essential for a candidate of the chosen competency level to have mastered, where the interpretation of 'essential' has first been discussed and agreed on, or ii) "important, but not essential". [40,41] The difficulty rating for each PSI will have already been estimated objectively by Rasch analysis from appropriate (or, calibrated) experimental data using the Rasch model. Thus, the mean or median difficulty rating in logits for each judge's "essential item group" can be obtained.…”

Development of Objective Standard Setting Using Rasch Measurement Model in Malaysian Institution of Higher Learning

“…Angoff had its way by averaging the subject-matter experts predicted difficulty of the items (James et al, 1998) in which described by Sick (2009) as "too difficult and confusing". Objective (OSS) model and Bookmark represent clearly departures from Angoff model, Ebeland Nedelsky (Stone, 2011). Compared with both model, Angoff failed to define a legitimate and stable construct and avoids the item by item judgement may be tedious and difficult to judges (Stone, 2011;MacCann & Stanley, 2006).…”