High-Stakes Examinations

Reznick, Richard K.; Regehr, Glenn; Yee, Gilbert; Rothman, Arthur I.; Blackmore, David; Dauphinee, Dale

doi:10.1097/00001888-199810000-00058

Cited by 42 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may seem counter-intuitive, as case-specific, task oriented item lists were thought to minimize inter-rater variability, but various studies have shown this is not the case [15] . Our results are consistent with research in other domains spanning many years which has shown that checklists are not as objective as originally supposed [16] , [17] , have inferior validity, and do not show better reliability than global rating scales [18] – [21] . The dimension ‘timely’ in the checklist showed a higher reliability than the dimension ‘correct’.…”

Section: Discussionsupporting

confidence: 92%

Assessing the Assessment in Emergency Care Training

Dankbaar

Stegers‐Jager

Baarveld³

et al. 2014

PLoS ONE

View full text Add to dashboard Cite

ObjectiveEach year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used formats in assessing emergency care skills.MethodsResidents were assessed at the end of a 2-week emergency course; a subgroup was videotaped. Psychometric analyses were conducted to assess the validity and inter-rater reliability of the assessment instrument, which included a checklist, a 9-item competency scale and a global performance scale.ResultsA group of 144 residents and 12 raters participated in the study; 22 residents were videotaped and re-assessed by 8 raters. The checklists showed limited validity and poor inter-rater reliability for the dimensions “correct” and “timely” (ICC = .30 and.39 resp.). The competency scale had good construct validity, consisting of a clinical and a communication subscale. The internal consistency of the (sub)scales was high (α = .93/.91/.86). The inter-rater reliability was moderate for the clinical competency subscale (.49) and the global performance scale (.50), but poor for the communication subscale (.27). A generalizability study showed that for a reliable assessment 5–13 raters are needed when using checklists, and four when using the clinical competency scale or the global performance scale.ConclusionsThis study shows poor validity and reliability for assessing emergency skills with checklists but good validity and moderate reliability with clinical competency or global performance scales. Involving more raters can improve the reliability substantially. Recommendations are made to improve this high stakes skill assessment.

show abstract

Section: Discussionsupporting

confidence: 92%

Assessing the Assessment in Emergency Care Training

Dankbaar

Stegers‐Jager

Baarveld³

et al. 2014

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Nevertheless, here are some limitations about this study. First, the OSCE results are only based on checklist grids whereas many studies showed that global rating should be more effective to assess competencies (Hodges et al, 1999;Reznick et al, 1998). A global score should be added to the next edition to compare this score to the checklist results.…”

Section: Discussionmentioning

confidence: 99%

Education and New Developments 2022 – Volume 2

2022

View full text Add to dashboard Cite

This book contains the full text of papers and posters presented at the International Conference on Education and New Developments (END 2022), organized by the World Institute for Advanced Research and Science (WIARS).Education, in our contemporary world, is a right since we are born. Every experience has a formative effect on the constitution of the human being, in the way one thinks, feels and acts. One of the most important contributions resides in what and how we learn through the improvement of educational processes, both in formal and informal settings. The International Conference seeks to provide some answers and explore the processes, actions, challenges and outcomes of learning, teaching and human development. The goal is to offer a worldwide connection between teachers, students, researchers and lecturers, from a wide range of academic fields, interested in exploring and giving their contribution in educational issues. We take pride in having been able to connect and bring together academics, scholars, practitioners and others interested in a field that is fertile in new perspectives, ideas and knowledge.We counted on an extensive variety of contributors and presenters, which can supplement our view of the human essence and behavior, showing the impact of their different personal, academic and cultural experiences. This is, certainly, one of the reasons we have many nationalities and cultures represented, inspiring multi-disciplinary collaborative links, fomenting intellectual encounter and development.END 2022 received 790 submissions, from more than 45 different countries, reviewed by a double-blind process. Submissions were prepared to take form of Oral Presentations, Posters, Virtual Presentations and Workshops. The conference accepted for presentation 263 submissions (33% acceptance rate), from which, 233 submissions are published in full text in these volumes.

show abstract

“…Current evidence suggests that the use of holistic scoring or global rating scales by an experienced physician shows greater inter-station reliability, better construct validity, and better concurrent validity compared to checklists [25]. Global rating scales allow the examiner to rate the whole process compared to rating scales looking at one aspect alone specially when assessing areas such as judgment, empathy, organization of knowledge and technical skills [26, 27]. For OSCEs which use the BRM for establishing a standard setting, the use of the two types of checklists is mandatory.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the validity evidence of an OSCE: results from a new medical school

et al. 2018

View full text Add to dashboard Cite

BackgroundTo prevent the problems of traditional clinical evaluation, the “Objective Structured Clinical Examination (OSCE)” was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.MethodsFifty-three first year medical students took part in a summative OSCE at the Lebanese American University-School of Medicine. Evidence to support consequential validity was gathered by using criterion-based standard setting methods. Internal structure validity evidence was gathered by examining various psychometric measures both at the station level and across the complete OSCE.ResultsCompared to our actual method of computing results, the introduction of standard setting resulted in lower students’ average grades and a higher cut score. Across stations, Cronbach’s alpha was moderately low.ConclusionGathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE. It is critical that this analysis be performed routinely on local iterations of given tests, and the results used to enhance the quality of assessment.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1421-x) contains supplementary material, which is available to authorized users.

show abstract

High-Stakes Examinations

Cited by 42 publications

References 0 publications

Assessing the Assessment in Emergency Care Training

Assessing the Assessment in Emergency Care Training

Education and New Developments 2022 – Volume 2

Evaluating the validity evidence of an OSCE: results from a new medical school

Contact Info

Product

Resources

About