What Are Panelists Thinking When They Participate in Standard-Setting Studies?

Skorupski, William P.; Hambleton, Ronald K.

doi:10.1207/s15324818ame1803_3

Cited by 29 publications

(23 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This effect has also been described before (Skorupski and Hambleton 2005;O'Donovan, Price, and Rust 2000). We incorporated the training session to impede it, but the current findings indicate that despite the training the understanding of the criteria differed.…”

Section: Conclusion and Discussionsupporting

confidence: 68%

“…This creates complex conditions for which existing standard-setting procedures are not adequately equipped. Equivocal understanding of criteria has been found to influence the outcomes of more traditional standard-setting procedures (Skorupski and Hambleton 2005). Price (2005) and Sadler (1987) argue that standards resulting from standard-setting procedures are contingent on the local situation and the standard-setting procedure selected, as all procedures rely on human judgement (Norcini and Shea 1997;Berk 1996).…”

Section: Standard Settingmentioning

confidence: 99%

“…For that reason a part of the training session was dedicated to the joint establishment of what the programme's CAP exactly entailed, enabling an analysis of the influence of the local situation in terms of the implemented CAP. Secondly, the participants needed a clear and correct understanding of the quality criteria (following Sadler 1998), as criteria used in many standard-setting studies can be open to multiple interpretations (Skorupski and Hambleton 2005). As we considered an adequate understanding of the quality criteria to be a prerequisite for the evaluation, the remainder of the session was devoted to the explanation of the CAP framework and its 12 quality criteria.…”

Section: Training Sessionmentioning

confidence: 99%

See 2 more Smart Citations

The explication of quality standards in self-evaluation

Bronkhorst

Baartman

Stokking

2012

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

Section: Conclusion and Discussionsupporting

confidence: 68%

Section: Standard Settingmentioning

confidence: 99%

Section: Training Sessionmentioning

confidence: 99%

See 1 more Smart Citation

The explication of quality standards in self-evaluation

Bronkhorst

Baartman

Stokking

2012

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

“…In rounds 2 and 3, participants were more likely to rely on the opinions of other panelists to guide their decision making. Recent research has shown panelists participating in standard‐setting studies using other methods consider these same factors (e.g., Ferdous & Plake, 2005; Skorupski & Hambleton, 2005).…”

Section: Review Of Research On Bookmarkmentioning

confidence: 99%

The Bookmark Standard‐Setting Method: A Literature Review

Karantonis

Sireci

2006

Educational Measurement

View full text Add to dashboard Cite

The Bookmark method for setting standards on educational tests is currently one of the most popular standard‐setting methods. However, research to support the method is scarce. In this report, we review the published and unpublished literature on this method as well as some seminal work in the area of evaluating standard‐setting studies. Our review highlights both strengths and limitations of the method. Strengths include its wide acceptance and panelist confidence in the method. Limitations include a potential bias to produce lower‐than‐intended standards and problems in selecting the most appropriate response probability value for ordering the items presented to panelists. It is clear that more research on this method is needed to support its wide use. Several areas for future research to better understand the validity of the Bookmark method for setting standards on educational tests are presented.

show abstract

“…This was confirmed and amplified by Margolis and Clauser (2014), who found that the use of performance information also resulted in a reduction of panelist cut score selection variability. Even with feedback, it is found that panelists may not understand the consequences of their own recommended standard (Skorupski & Hambleton, 2005; Skorupski, 2012).…”

Section: Introductionmentioning

confidence: 99%

Adding Objectivity to Standard Setting: Evaluating Consequence Using the Conscious and Subconscious Weight Methods

Leventhal

Grabovsky

2020

Educational Measurement

View full text Add to dashboard Cite

Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing remediation has high‐impact consequences for students. Unfortunately, tests that classify individuals are subjected to false positive and false negative misclassifications. When determining a standard, standard setting panelists implicitly consider the negative consequences of the decisions made from test use. We propose the conscious weight method and subconscious weight method to bring more objectivity to the standard setting process. To do this, these methods quantify the relative harm of the negative consequences of false positive and false negative misclassification.

show abstract

What Are Panelists Thinking When They Participate in Standard-Setting Studies?

Cited by 29 publications

References 6 publications

The explication of quality standards in self-evaluation

The explication of quality standards in self-evaluation

The Bookmark Standard‐Setting Method: A Literature Review

Adding Objectivity to Standard Setting: Evaluating Consequence Using the Conscious and Subconscious Weight Methods

Contact Info

Product

Resources

About