Strategies in Comparison of Methods for Scoring Patient Managementproblems

Webster, George D.; Shea, Judy A.; Norcini, John J.; Grosso, Louis J.; Swanson, David B.

doi:10.1177/016327878801100206

Cited by 15 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This may appear contradictory (and it is not true for all cases); however, it is essential that these iteins have a negligible, if not negative, weight if the efficiency of the expert is to be appropriately distinguished from the thoroughness of the less advanced student (or the floundering of the novice). Weaknesses in this aspect of scoring have been described as problematic in the context of paper-and-pencil assessments of clinical competence known as "patient management problems" (Webster, Shea, Norcini, Grosso, & Swanson, 1988). Webster et al reported that efficiency represented by the ratio of indicated actions to all actions selected by an examinee was superior to a more complex proficiency score for discriminating between examinees from high-and low-ability groups.…”

Section: Discussionmentioning

confidence: 99%

Scoring a Performance‐Based Assessment by Modeling the Judgments of Experts

Clauser

Subhiyah

Nungester

et al. 1995

J Educational Measurement

View full text Add to dashboard Cite

Performance assessments typically require expert judges to individually rate each performance. This results in a limitation in the use of such assessments because the rating process may be extremely time consuming. This article describes a scoring algorithm that is based on expert judgments but requires the rating of only a sample of performances. A regression-based policy capturing procedure was implemented to model the judgment policies of experts. The data set was a seven-case performance assessment of physician patient management skills. The assessment used a computer-based simulation of the patient care environment. The results showed a substantial improvement in correspondence between scores produced using the algorithm and actual ratings, when compared to raw scores. Scores based on the algorithm were also shown to be superior to raw scores and equal to expert ratings for making pass~fail decisions which agreed with those made by an independent committee of experts.Among the criteria identified by Linn, Baker, and Dunbar (1991) for evaluating performance-based assessments are the requirements that they be efficient (and cost effective) and that the results have sufficient generalizability. Many (perhaps most) performance assessments require experts to judge the individual performances or the durable products of those performances. The use of judges can impact significantly on both efficiency and generalizability. Almost by definition, the need for judges reduces efficiency. As Wainer and Thissen (1993) have graphically demonstrated for large-scale assessments, where the alternative is likely to be mechanical scoring, assessments scored by judges will be more expensive even when relatively few judgments are required, and the discrepancy in cost will increase dramatically as the required number of judgments increases.The judge facet may also contribute substantially to error variance and so

show abstract

Section: Discussionmentioning

confidence: 99%

Scoring a Performance‐Based Assessment by Modeling the Judgments of Experts

Clauser

Subhiyah

Nungester

et al. 1995

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Existing findings showed that the use of different weighting of case test items or options, depending on their importance and appropriateness to the assessed task, has little impact on the resulting scores (Stillman et al, 1986b). In addition, empirical or expert-derived scoring, such as aggregate scoring, where the score of an item is proportional to the degree of agreement between experts (Norman, 1985), is found to correlate with other type of scoring and neither affects nor improves the validity of test scores (Webster, Shea, Norcini, Grosso, & Swanson, 1988). Finally, preliminary results showed that a scoring that takes into account both the examinees' performance outcomes and their underlying reasoning seems to be more discriminating than one that relies solely on the outcomes (Vu, Lee, & Steward, 1990).…”

Section: Test Scoring Validitymentioning

confidence: 99%

Use of Standardized Patients in Clinical Assessments: Recent Developments and Measurement Findings

Barrows

1994

Educational Researcher

115

View full text Add to dashboard Cite

This article reviews recent developments and measurement findings on the use of live patient simulations or "standardized patients" in performance examinations to assess the competence of medical professionals. The results of large-scale standardized patient-based performance assessments are presented and discussed in terms of their feasibility, reliability, validity, and implications for assessing competence in other professions.

show abstract

“…Als over een optie geen consensus kan worden bereikt, wordt deze optie uit de lijst verwijderd. Een veel gehanteerde indeling wordt hieronder weergegeven (Norcini et al, 1983;Webster et al, 1988). Achter elke categorie staat de bijbehorende wegingsfactor vermeld, zoals gebruikt bij de American Board of Internal Medicine.…”

Section: Schriftelijke Simulatie-instrumenten: Een Overzichtunclassified

“…Op basis van deze ruwe scores kunnen verschillende, meer classificerende scores worden bepaald. De meest gebruikte scores zijn de "proficiency score" en de "efficiency score", (McGuire & Babbott, 1967;Webster et al, 1988).…”

Section: Schriftelijke Simulatie-instrumenten: Een Overzichtunclassified

“…Op deze wijze kan beter inzicht worden verkregen in hetgeen de student daadwerkelijk weet en wordt gokgedrag tegengegaan. De in de literatuur voorkomende benamingen voor deze scores zijn: select vs. omit, throughness vs. non-select en errors-of-omission vs. commission (Webster et al, 1988). Uitgaande van bovenstaande categorieën, worden beide scores als volgt berekend:…”

Section: Efficiencyunclassified

See 1 more Smart Citation

Over structurering van beoordelingsmethoden voor open vragen

Frijns¹

View full text Add to dashboard Cite

People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

show abstract

Strategies in Comparison of Methods for Scoring Patient Managementproblems

Cited by 15 publications

References 7 publications

Scoring a Performance‐Based Assessment by Modeling the Judgments of Experts

Scoring a Performance‐Based Assessment by Modeling the Judgments of Experts

Use of Standardized Patients in Clinical Assessments: Recent Developments and Measurement Findings

Over structurering van beoordelingsmethoden voor open vragen

Contact Info

Product

Resources

About