Too hot to handle? Assessing the validity and reliability of the College of Intensive Care Medicine "Hot Case” examination

Cohen, Jérémy; Pinder, Mary; Angelico, Daniel; Keo, Frieda

doi:10.51893/2022.1.l

Cited by 3 publications

(1 citation statement)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, and of great interest to College Fellows, this issue of CCR hosts an important exchange of views on the clinical exam (the so called "hot cases”) and how it can be improved. 12 , 13 Such exchange would reassure current and would-be fellows that the College remains committed to the highest standards of training and assessment and to a continuing improvement approach to such goals.…”

mentioning

confidence: 99%

In This Issue of CCR

Bellomo¹

2022

Critical Care and Resuscitation

View full text Add to dashboard Cite

show abstract

mentioning

confidence: 99%

In This Issue of CCR

Bellomo¹

2022

Critical Care and Resuscitation

View full text Add to dashboard Cite

show abstract

The reliability of the College of Intensive Care Medicine of Australia and New Zealand “Hot Case” examination

Hoffman,

Swanson,

Lane

et al. 2024

BMC Med Educ

View full text Add to dashboard Cite

Background High stakes examinations used to credential trainees for independent specialist practice should be evaluated periodically to ensure defensible decisions are made. This study aims to quantify the College of Intensive Care Medicine of Australia and New Zealand (CICM) Hot Case reliability coefficient and evaluate contributions to variance from candidates, cases and examiners. Methods This retrospective, de-identified analysis of CICM examination data used descriptive statistics and generalisability theory to evaluate the reliability of the Hot Case examination component. Decision studies were used to project generalisability coefficients for alternate examination designs. Results Examination results from 2019 to 2022 included 592 Hot Cases, totalling 1184 individual examiner scores. The mean examiner Hot Case score was 5.17 (standard deviation 1.65). The correlation between candidates’ two Hot Case scores was low (0.30). The overall reliability coefficient for the Hot Case component consisting of two cases observed by two separate pairs of examiners was 0.42. Sources of variance included candidate proficiency (25%), case difficulty and case specificity (63.4%), examiner stringency (3.5%) and other error (8.2%). To achieve a reliability coefficient of > 0.8 a candidate would need to perform 11 Hot Cases observed by two examiners. Conclusion The reliability coefficient for the Hot Case component of the CICM second part examination is below the generally accepted value for a high stakes examination. Modifications to case selection and introduction of a clear scoring rubric to mitigate the effects of variation in case difficulty may be helpful. Increasing the number of cases and overall assessment time appears to be the best way to increase the overall reliability. Further research is required to assess the combined reliability of the Hot Case and viva components.

show abstract

Untitled

2022

CC&R

View full text Add to dashboard Cite

Too hot to handle? Assessing the validity and reliability of the College of Intensive Care Medicine "Hot Case” examination

Cited by 3 publications

References 1 publication

In This Issue of CCR

In This Issue of CCR

The reliability of the College of Intensive Care Medicine of Australia and New Zealand “Hot Case” examination

Untitled

Contact Info

Product

Resources

About