An application of Multifaceted Rasch measurement in the Yes/No Angoff standard setting procedure

Abstract

When implementing standard setting procedures, there are two major concerns: variance between panelists and efficiency in conducting multiple rounds of judgments. With regard to the former, there is concern over the consistency of the cutoff scores made by different panelists. If the cut scores show an inordinately wide range then further rounds of group discussion are required to reach consensus, which in turn leads to the latter concern. The Yes/No Angoff procedure is typically implemented across several rounds. Panelists revise their original decisions for each item based on discussion with co-panelists between each round. The purpose of this paper is to demonstrate a framework for evaluating the judgments in the standard setting process. The Multifaceted Rasch model was applied as a tool to evaluate the quality of standard setting in a context of language assessment. The results indicate that the Multifaceted Rasch model offers a promising approach to examination of the variability in the standard setting procedures. In addition, this model can identify aberrant decision making for each panelist, which can be used as feedback for both standard setting designers and panelists.

Standard Setting in Academic Writing Assessment through Objective Standard Setting Method

“…Table 5 shows the comparative results of two-step cluster analysis and OSS Method. (Hsieh, 2013). It has important consequences for the stakeholders such as students, teachers, and policy-makers in different areas (Fulcher, 2013;Shin & Lidster, 2017;Sondergeld et al, 2020;Stone et al, 2011).…”

Development and Validation of a Diagnostic Rating Scale for EFL Writing in China

“…Validation of a rating scale is needed since it is considered a fundamental concern of testing (Messick, 1989) and is associated with fairness and social responsibilities of a test (Kane, 2010). Multi-facet Rasch model analysis (MFRM) and generalizing theory (G-theory) are two mainstream validating approaches in recent studies (Beglar, 2009;Hsieh, 2013;Bochner, 2015). Through comparison, a multi-facet Rasch analysis is selected as the specific method for validation, as MFRM provides finer and more individual analyses and performs better in analyzing how test bias occurs than G-theory does (Kim & Wilson, 2009;McNamara & Knoch, 2012;Sudweeks, Reeve, & Bradshaw, 2004).…”

Evaluating different standard-setting methods in an ESL placement testing context

“…There have been relatively few standard-setting studies in language-testing contexts. Several studies have, for example, linked standardized test scores to levels of the Common European Framework of Reference (CEFR) (Bechger, Kuijper, & Maris, 2009;Papageorgiou, 2010), determined IELTS® test cutscores for nursing program admission (O'Neill, Buckendahl, Plake, & Taylor, 2007), and set cutscores to match performance descriptors of the sixth-grade national English curriculum in Taiwan (Hsieh, 2013a(Hsieh, , 2013b. Despite the burgeoning need for guidance in establishing placement cutscores in English for academic purposes (EAP) or other language programs, there has, to our knowledge, been only one standard-setting study conducted in a language placement testing context: Shin (2004) used three approaches -the Angoff method, the Borderline group method, and cluster analysis -to determine cutoffs for a newly developed webbased English placement exam used in an advanced academic ESL program.…”

“…Both approaches thus have different strengths and weaknesses, which have led some researchers to call for using both together (Green, Trimble, & Lewis, 2003). However, perhaps unsurprisingly given the different focuses of test-centered and examinee-centered methods, the results of different standard-setting procedures usually differ, often by large margins (Hsieh, 2013a; Jaeger, 1989; Shin, 2004), and scholars do not agree on how to choose among the different standard-setting methods in educational assessment (Zieky, Perie, & Livingston, 2008). Indeed, although many different standard-setting methods have been compared and evaluated in a number of previous studies in the field of educational measurement (Zieky, 2001), these studies did not find evidence of convergence between resulting cutscores, nor did they provide support for the superiority of a single method (Alsmadi, 2007; Kane, 1994, 2001; Livingstone & Zieky, 1989; Näsström & Nyström, 2008).…”