The standard setting process: validating interpretations of stakeholders

Abstract

Background: Stakeholders' interpretations of the findings of large-scale educational assessments can influence important decisions. In the context of educational assessment, standard-setting remains an especially critical element, because it is complex and largely unstandardized. Instruments established by means of standard-setting procedures such as proficiency levels (PL) therefore appear to be arbitrary to some degree. Owing to the significance such results take on, when they are communicated to stakeholders or the public, a thorough validation of this process seems crucial. In our study, ministry stakeholders intended to use PL established in an assessment of science abilities to obtain information about students' strengths and weaknesses regarding science abilities in general and specifically about the extent to which they were prepared for future science studies. The aim of our study was to investigate the validity arguments regarding these two intended interpretations.Methods: Based on a university science test administered to 3641 upper secondary students (Grade 13), a panel of nine experts set four cut scores using two variations of the Angoff method, the Yes/No Angoff method (multiple choice items) and the extended Angoff method (complex multiple choice items). We carried out t-tests, repeated measures ANOVA, G-studies and regression analyses to support the procedural, internal, external, and consequential validity elements regarding the aforementioned interpretations of the cut scores.Results: Our t-tests and G-studies showed that the intended use of the cut scores was valid regarding procedural and internal aspects of validity. These findings were called into question by the experts' lack of confidence in the established cut scores. Regression analyses including number of lessons taught and intended and pursued sciencerelated studies showed good external and poor consequential validity. Conclusion:The cut scores can be used as an indicator of 13th graders' strengths and weaknesses in science. They should not be used as an indicator for preparedness for science university studies. Since assessment formats are continually evolving and consequently leading to more complex designs, further research needs to be conducted on the application of new standard-setting methods to meet the challenges arising from this development. which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests

“…Meanwhile, some studies, including the study conducted by Yim [ 35 ], external validity is evaluated through comparing the results of other standard-setting procedures. Although this is common and confirmed in the sources [ 39 , 40 ], but when intended to compare the external validity of two standard-setting procedures, it is better to consider an external criterion [ 41 ], due to implementation problems, but it’s less commonly used [ 40 ]. This is closer to Kane’s definition of external validity [ 5 ].…”

Developing a scoring framework for curriculum viability indicators to assess the survival of a curriculum

“…(3). Tertile categorization was done to determine the cut off range (12). The tertile categorization is a method of dividing a data set into three equal parts with two cut-off points (e.g., with quartiles 1 and 3): one category in the middle and two on the tails.…”

A Discovery on Malaysian Esl Teachers’ Adherence to Policy Mandates in Classroombased Assessment Practices

“…According to Wills et al ( 2016), standard setting is a practice where teachers share and develop their understanding of what learning looks like by comparing various types and levels of student work to formal standards and success criteria. According to Kampa et al (2019), the term standard-setting encompasses a variety of consensual approaches that involve teachers setting discrete cut scores and comparing them to proficiency scales. In brief, it is the practice of identifying the aspects or criteria of classroom assessment that can assist teachers in exercising their professional judgement.…”