Comparing Yes/No Angoff and Bookmark Standard Setting Methods in the Context of English Assessment

Hsieh, Ming-Chuan

doi:10.1080/15434303.2013.769550

Cited by 10 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bookmark approaches to assess language proficiency levels of students [26]. Results of this study showed that the mean scores obtained from 32 experts, using these two methods, were different in determining the three final cut-scores.…”

Section: Several Studies Have Compared the Angoff And Bookmark Approamentioning

confidence: 79%

Comparison of validity of Bookmark and Angoff Standard Setting Methods in Medical performance tests

Afrashteh¹

2020

Preprint

View full text Add to dashboard Cite

Introduction One of the main processes in evaluating of the students’ performance is standard staging to determine the passage for the test. The purpose of this study was to compare the validity of two methods of Angoff and bookmark in standard setting. Method Participants included 190 master’s students graduated in laboratory sciences since past year. Designed by a group of experts, a performance test with 32 item was used in this study to assess laboratory skills of graduates of medical laboratory sciences. Moreover, two groups of experts voluntarily participated in this study to set the cut-score. To assess the process validity, a 5-item questionnaire was asked from two groups of penists. To investigate the internal validity, the variance of the cut scores determined by the members of the two panels was compared with the F ratio. External validity was assessed by using four indices of correlation test with criterion score. Results Comparison of the two methods of Angoff and bookmarking showed that the mean of process validity indices was higher in bookmarking method. In order to assess internal validity, conclusion: Homogeneity of results and co-ordination of judges' scores were considered. Conclusion In evaluating of the external validity (concordance of the cut score with the criterion score), All five external validity indices supported the bookmark method.

show abstract

Section: Several Studies Have Compared the Angoff And Bookmark Approamentioning

confidence: 79%

Comparison of validity of Bookmark and Angoff Standard Setting Methods in Medical performance tests

Afrashteh¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In our study, we chose the Angoff method, because it can easily be applied to different item formats of a test (Plake and Cizek 2012) and showed similar results compared to other (test-centered) standard-setting methods (e.g. ;Hsieh 2013;Skaggs and Hein 2011;Yudkowski et al 2008). Any choice of a standard-setting method leads to advantages and disadvantages, which we have laid out in the theoretical part of this article.…”

Section: Discussionmentioning

confidence: 99%

“…This criticism led to a considerable number of comparison studies between the Angoff method variations and other mostly test-centered standard setting methods as well as to the development of the Yes/No method (e.g. ;Hsieh 2013;Skaggs and Hein 2011;Yudkowski et al 2008). In these comparison studies the (Yes/No) Angoff method has either been favored or showed no substantial differences to the other methods.…”

Section: The Angoff Methods and Two Of Its Variationsmentioning

confidence: 99%

“…Additionally, the Yes/No method has been favored by judges over the original Angoff method in standard-settings, because a dichotomous judgement seems to be easier for them than estimating percentages, which is the task within the original Angoff method (for the original method, Angoff 1971; for recent studies on the Yes/No method e.g., Yudkowski et al 2008;Hsieh 2013). However, this simplification for the judges leads to a loss of information and cut score judgments tend to move from the center to the extremes (Hsieh 2013). Imagine a test that only consists of items a minimally competent student at PL1 would answer with a probability of 10%, at PL3 with a probability of 55%, at PL4 with a probability of 85%.…”

Section: The Angoff Methods and Two Of Its Variationsmentioning

confidence: 99%

“…Nevertheless, the described mental game does not reflect the complex reality of a standard-setting procedure. It is more likely that the items within one given PL will vary in difficulty and the proposed mental game is not a realistic expectation for any set of test items (Hsieh 2013). Therefore the Yes/No method as well as probability-based methods will produce suitable cut-scores.…”

Section: The Angoff Methods and Two Of Its Variationsmentioning

confidence: 99%

See 2 more Smart Citations

The standard setting process: validating interpretations of stakeholders

Kampa

Wagner

Köller

2019

Large-scale Assess Educ

View full text Add to dashboard Cite

Background: Stakeholders' interpretations of the findings of large-scale educational assessments can influence important decisions. In the context of educational assessment, standard-setting remains an especially critical element, because it is complex and largely unstandardized. Instruments established by means of standard-setting procedures such as proficiency levels (PL) therefore appear to be arbitrary to some degree. Owing to the significance such results take on, when they are communicated to stakeholders or the public, a thorough validation of this process seems crucial. In our study, ministry stakeholders intended to use PL established in an assessment of science abilities to obtain information about students' strengths and weaknesses regarding science abilities in general and specifically about the extent to which they were prepared for future science studies. The aim of our study was to investigate the validity arguments regarding these two intended interpretations.Methods: Based on a university science test administered to 3641 upper secondary students (Grade 13), a panel of nine experts set four cut scores using two variations of the Angoff method, the Yes/No Angoff method (multiple choice items) and the extended Angoff method (complex multiple choice items). We carried out t-tests, repeated measures ANOVA, G-studies and regression analyses to support the procedural, internal, external, and consequential validity elements regarding the aforementioned interpretations of the cut scores.Results: Our t-tests and G-studies showed that the intended use of the cut scores was valid regarding procedural and internal aspects of validity. These findings were called into question by the experts' lack of confidence in the established cut scores. Regression analyses including number of lessons taught and intended and pursued sciencerelated studies showed good external and poor consequential validity. Conclusion:The cut scores can be used as an indicator of 13th graders' strengths and weaknesses in science. They should not be used as an indicator for preparedness for science university studies. Since assessment formats are continually evolving and consequently leading to more complex designs, further research needs to be conducted on the application of new standard-setting methods to meet the challenges arising from this development. which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

show abstract

Standard Setting in a Formative Assessment of Digital Responsibility Among Norwegian Eighth Graders

Hatlevik

Radtke

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

Comparing Yes/No Angoff and Bookmark Standard Setting Methods in the Context of English Assessment

Cited by 10 publications

References 21 publications

Comparison of validity of Bookmark and Angoff Standard Setting Methods in Medical performance tests

Comparison of validity of Bookmark and Angoff Standard Setting Methods in Medical performance tests

The standard setting process: validating interpretations of stakeholders

Standard Setting in a Formative Assessment of Digital Responsibility Among Norwegian Eighth Graders

Contact Info

Product

Resources

About