Examining the Costs of Performance Assessment

Hardy, Roy A.

doi:10.1207/s15324818ame0802_2

Cited by 18 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The cost of scoring has been identified as a major limitation for large-scale implementation of performance assessments (Hardy, 1995;Wainer & Thissen, 1993). The reason for the substantial disparity between the cost of scoring multiple-choice items and the cost of scoring performance tasks is that the former can be optically scanned by a machine, whereas the latter are typically individually reviewed and rated by experts.…”

mentioning

confidence: 99%

Development of Automated Scoring Algorithms for Complex Performance Assessments: A Comparison of Two Approaches

Clauser

Margolis

Clyman

et al. 1997

J Educational Measurement

View full text Add to dashboard Cite

Performance assessments are typically scored by having experts rate individual performances. The cost associated with using expert raters may represent a serious limitation in many large‐scale testing programs. The use of raters may also introduce an additional source of error into the assessment. These limitations have motivated development of automated scoring systems for performance assessments. Preliminary research has shown these systems to have application across a variety of tasks ranging from simple mathematics to architectural problem solving. This study extends research on automated scoring by comparing alternative automated systems for scoring a computer simulation test of physicians'patient management skills; one system uses regression‐derived weights for components of the performance, the other uses complex rules to map performances into score levels. The procedures are evaluated by comparing the resulting scores to expert ratings of the same performances.

show abstract

mentioning

confidence: 99%

Development of Automated Scoring Algorithms for Complex Performance Assessments: A Comparison of Two Approaches

Clauser

Margolis

Clyman

et al. 1997

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…(Baker & Brown, in press). For example, in the area of essay scoring, the cost of evaluating thousands of essays has led to the selection of testing approaches that can be scored efficiently, such as multiple-choice formats (Hardy, 1995). In our case, the computer provides the power to score student maps directly and efficiently.…”

Section: Example Flowmentioning

confidence: 99%

Knowledge mapper authoring system prototype

Chung¹,

Baker²,

Cheak³

2002

PsycEXTRA Dataset

View full text Add to dashboard Cite

We have developed a prototype authoring system for knowledge mapping that provides the capability to (a) specify map tasks (i.e., specify terms and links), (b) launch a knowledge map related to a task, (c) specify criterion or expert maps (for scoring purposes), (d) define groups of users and associate tasks to those groups, and (e) publish tasks for global access. The prototype interface is a simple HTML interface, with any destination function accessible within four mouse clicks. A usability study was conducted with one teacher and 62 fourth-and fifth-grade students. Students were considered beta testers, and the teacher, who had used our existing knowledge mapper, was considered an expert. The teacher reported using the authoring system to create pretest and posttest tasks for students and to access and evaluate student maps. Students used the authoring system to create knowledge maps of their group research projects. The teacher reported that the authoring system was easy to use and thought student authoring promoted ownership of work. Students using the authoring system created more sophisticated knowledge maps, especially in the links specified by students. This anecdotal report is bolstered by analyses of the links used across all student-authored maps. Thirty-eight percent of the links were causal (e.g., causes) or functional (e.g., protects), and 24% were part-whole (e.g., part of). Limited screen real estate, confusing map access, complicated login functionality, and system crashes limited usability. Student performance on a usability task showed that, in general, students were able to independently carry out major authoring and mapping tasks; however, a positive relationship was found between the amount of time a student was at the keyboard and the amount of help that student needed during the usability task, suggesting that practice using the authoring system improves fluency with the authoring system. 1 We wish to acknowledge and thank Joanne Michiuye and Linda de Vries of UCLA/CRESST for their help during data collection, and Ali Abedi of UCLA/CRESST for programming and technical support. We also wish to thank Terri Robinson for her content expertise. We also thank Jan Cohen, Margaret Heritage, and Sharon Sutton at UCLA's Seeds University Elementary School (UES) for their assistance, participation, insights, and support. And finally we are deeply grateful to the creative and gifted teacher and her students at UES who participated and assisted in this research study.

show abstract

“…Highly reliable ratings are possible even in large-scale testing programs, but they typically come at considerable cost. This limiting factor in the use of raters has recently been examined in some detail (Hardy, 1995;Wainer & Thissen, 1993), but it is not a new issue (e.g., Coffman, 1971;Stalnaker, 1951). One approach to eliminating rater error-as well as the associated cost-is to replace the rater with an automated scoring system.…”

Section: Automated Scoringmentioning

confidence: 99%

Recurrent Issues and Recent Advances in Scoring Performance Assessments

Clauser

2000

Applied Psychological Measurement

View full text Add to dashboard Cite

A conceptual framework is provided to guide the development of scoring procedures for performance assessments. Within this framework are four questions: (1) what aspects of the performance are to be scored, (2) what criteria are to be applied to evaluate the identified aspects or components of the performance, (3) how should the scoring criteria be developed, and (4) how should these criteria be applied? The historical limitations of performance assessments are reviewed and recent efforts to avoid those limitations are discussed in the context of these questions.

show abstract

Examining the Costs of Performance Assessment

Cited by 18 publications

References 3 publications

Development of Automated Scoring Algorithms for Complex Performance Assessments: A Comparison of Two Approaches

Development of Automated Scoring Algorithms for Complex Performance Assessments: A Comparison of Two Approaches

Knowledge mapper authoring system prototype

Recurrent Issues and Recent Advances in Scoring Performance Assessments

Contact Info

Product

Resources

About