Development of a Scoring Algorithm to Replace Expert Rating for Scoring a Complex Performance-Based Assessment

Clauser, Brian E.; Ross, Linette P.; Clyman, Stephen G.; Rose, Kathie M.; Margolis, Melissa J.; Nungester, Ronald J.; Piemme, Thomas E.; Chang, Lucy; El-Bayoumi, Gigi; Malakoff, Gary L.; Pincetl, Pierre

doi:10.1207/s15324818ame1004_3

Cited by 25 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When they are introduced into the medical licensure examination, the simulations will be scored using an automated system. This system has been described in detail previously (Ciauser et al, 1995;Clauser et al, 1997). The automated scoring system represents an explicit effort to model policies used by expert raters in scoring examinee performances.…”

Section: Methodsmentioning

confidence: 99%

Components of Rater Error in a Complex Performance Assessment

Clauser

Clyman

Swanson

1999

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

Numerous studies have examined performance assessment data using generaliz‐ability theory. Typically, these studies have treated raters as randomly sampled from a population, with each rater judging a given performance on a single occasion. This paper presents two studies that focus on aspects of the rating process that are not explicitly accounted for in this typical design. The first study makes explicit the “committee” facet, acknowledging that raters often work within groups. The second study makes explicit the “rating‐occasion” facet by having each rater judge each performance on two separate occasions. The results of the first study highlight the importance of clearly specifying the relevant facets of the universe of interest. Failing to include the committee facet led to an overly optimistic estimate of the precision of the measurement procedure. By contrast, failing to include the rating‐occasion facet, in the second study, had minimal impact on the estimated error variance.

show abstract

Section: Methodsmentioning

confidence: 99%

Components of Rater Error in a Complex Performance Assessment

Clauser

Clyman

Swanson

1999

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…The NetPass prototype used logical rules to identify the presence or absence of key features of the interaction, systematicity of steps, and number and seriousness of errors . Clauser et al (1997) describe this style of automated scoring for interactive problem solving in simulated patientmanagement problems at the National Board of Medical Examiners. Viewing the interaction between an engineer and a network as a conversation carried out in the Cisco IOS language, DeMark and Behrens (2004) took a statistical language processing approach to analyzing the log files, with promising results in classifying learners along a novice-to-expert curriculum.…”

Section: Using Krs To Create Tasks and Manage Assessment Systemsmentioning

confidence: 99%

On the Roles of External Knowledge Representations in Assessment Design

Mislevy

Behrens

Bennett³

et al. 2007

PsycEXTRA Dataset

View full text Add to dashboard Cite

Abstract:People use external knowledge representations (KRs) to create, identify, depict, transform, store, share, and archive information. Learning to work with KRs is central to becoming proficient in virtually every discipline. As such, KRs play central roles in curriculum, instruction, and assessment. We describe five key roles of KRs in assessment:1. An assessment is itself a KR, which makes explicit the knowledge that is valued, ways it is used, and standards of good work.2. The analysis of any domain in which learning is to be assessed must include the identification and analysis of the KRs in that domain.3. Assessment tasks can be structured around the knowledge, relationships, and uses of domain KRs.4. "Design KRs" can be created to organize knowledge about a domain in forms that support the design of assessment.5. KRs in the discipline of assessment design can guide and structure domain analyses (re #2), task construction (re #3), and the creation and use of design KRs (re #4).The third and fourth roles are developed in greater detail, through an "evidence-centered" design perspective that reflects the fifth role. Recurring implications of technology that leverage the impact of KRs in assessment are highlighted, including task design supports and automated task construction and scoring. Ideas are illustrated with "generate examples" tasks and simulation tasks for computer network design and troubleshooting.

show abstract

“…Mills (2000) and Zenisky and Sireci (in press) provided comprehensive overviews of the variety of item formats associated with CBT. Examples of innovative CBT items include drag-and-drop items (Fitzgerald, 1998), problem-solving vignettes (Bejar, 1991;Martinez & Ben-nett, 1992), sequential problem solving (Clauser et al, 1995(Clauser et al, , 1997Drasgow, Olson-Buchanan, & Moberg, 1999), and generating examples .…”

Section: Innovative Item Formatsmentioning

confidence: 99%

Validity Issues in Computer‐Based Testing

Huff

Sireci²

2001

Educational Measurement

View full text Add to dashboard Cite

Advances in technology are stimulating the development of complex, computerized assessments. The prevailing rationales for developing computer‐based assessments are improved measurement and increased efficiency. In the midst of this measurement revolution, test developers and evaluators must revisit the notion of validity. In this article, we discuss the potential positive and negative effects computer‐based testing could have on validity, review the literature regarding validation perspectives in computer‐based testing, and provide suggestions regarding how to evaluate the contributions of computer‐based testing to more valid measurement practices. We conclude that computer‐based testing shows great promise for enhancing validity, but at this juncture, it remains equivocal whether technological innovations in assessment have led to more valid measurement.

show abstract

Development of a Scoring Algorithm to Replace Expert Rating for Scoring a Complex Performance-Based Assessment

Cited by 25 publications

References 12 publications

Components of Rater Error in a Complex Performance Assessment

Components of Rater Error in a Complex Performance Assessment

On the Roles of External Knowledge Representations in Assessment Design

Validity Issues in Computer‐Based Testing

Contact Info

Product

Resources

About