Validity of three clinical performance assessments of internal medicine clerks

Hull, Alan L.; Hodder, Sally; Berger, Bruce A.; Ginsberg, David; Lindheim, Nora; Quan, Janice; Kleinhenz, Mary Ellen

doi:10.1097/00001888-199506000-00013

Cited by 56 publications

(45 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In other words, a student's ability to perform during the process of sitting a written work versus in an OSCE station in front of an examiner on another human being could, by itself, provide the rationale for the low correlations seen between these two variables. Interestingly, the correlations between anatomy OSCE and written examinations (which were predominantly MCQs) from this study compare closely with the published literature on correlations between OSCE marks in general and MCQs in other subjects (Hull et al, 1995;Wass et al, 2001;Auewarakul et al, 2005). The ranges of correlations between these three different studies were 0.28-0.48, which are in a similar range and consistent with the findings obtained in this study.…”

Section: Anatomy Osce and Anatomy Written Examination Performance Corsupporting

confidence: 91%

The Anatomy Competence Score—A new marker for anatomical ability

Schoeman¹,

Chandratilake²

2011

Anatomical Sciences Ed

View full text Add to dashboard Cite

The assessment of students' ability in gross anatomy is a complex process as it involves the measurement of multiple facets. In this work, the authors developed and introduced the Anatomy Competence Score (ACS), which incorporates the three domains of anatomy teaching and assessment namely: theoretical knowledge, practical 3D application of the knowledge, and clinical or bedside application of knowledge on patients. Equal contributions from these tripartite domains were used to synthesize the ACS. The theory knowledge was assessed using MCQs and short answer questions while the knowledge of practical 3D application was assessed using an Objective Structured Practical Examination (OSPE). The clinical or bedside application of anatomy knowledge was assessed by an Objective Structured Clinical Examination (OSCE). In this correlation study, the authors examined the interdomain correlations of the summative marks for the three contributing domains of the ACS, in order to examine the rationality of this new marker. Three cohorts of medical students (n = 538) at St. George's, University of London (SGUL) Medical School in the United Kingdom were included and analyzed. The results demonstrated that the correlations between the three domains were significantly low or moderate. The three domains probably represent unique knowledge and abilities. Therefore, it would appear that the average of the domains scores (the ACS) provide a comprehensive picture of a student's ability in anatomy.

show abstract

Section: Anatomy Osce and Anatomy Written Examination Performance Corsupporting

confidence: 91%

The Anatomy Competence Score—A new marker for anatomical ability

Schoeman¹,

Chandratilake²

2011

Anatomical Sciences Ed

View full text Add to dashboard Cite

show abstract

“…[5][6][7][8] Similar correlations are found in the adult literature. 18,19 In their review of tests using standardized patients, van der Vleuten and Swanson 16 statistically correct for measurement error by making the tests perfectly reliable. In all cases the true correlations between other measures of competence, both written tests and clinical evaluations, increased greatly over the observed correlations, with the majority of coefficients greater than 0.40.…”

Section: Validity Of the Oscementioning

confidence: 99%

The Objective Structured Clinical Examination

Carraccio¹,

Englander²

2000

Arch Pediatr Adolesc Med

158

View full text Add to dashboard Cite

“…The reliability of similar types of assessment processes have been variable, although many have claimed adequate reliability (Keck and Arnold 1979;Kwolek et al 1997;Magzoub et al 1998;Kreiter et al 1998;Nasca et al 2002;Durning et al 2005;Beckman et al 2006;Cohen et al 2009;Kreiter et al 1998), others have been either equivocal (Cowles and Kubany 1959;Hull et al 1995;Schwanz et al 1995;Williams et al 2004), or found the reliability not acceptable (Levine and McGuire 1971;Davis et al 1986;Thompson et al 1990;Metheny 1991;Ryan et al 1996;Pulito et al 2007;Searle 2008). A common problem with many of the studies claiming reliability for this form of competency assessment has been the inappropriate use of the alpha coefficient for nested and/or unbalanced designs which appears to be common for workplace-based assessments (Keck and Arnold 1979;Magzoub et al 1998;Nasca et al 2002;Durning et al 2005;Cohen et al 2009).…”

Section: Comparison To Other Studiesmentioning

confidence: 99%

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

McGill

Vleuten

Clarke

2011

Adv in Health Sci Educ

View full text Add to dashboard Cite

Even though rater-based judgements of clinical competence are widely used, they are context sensitive and vary between individuals and institutions. To deal adequately with rater-judgement unreliability, evaluating the reliability of workplace rater-based assessments in the local context is essential. Using such an approach, the primary intention of this study was to identify the trainee score variation around supervisor ratings, identify sampling number needs of workplace assessments for certification of competence and position the findings within the known literature. This reliability study of workplace-based supervisors' assessments of trainees has a rater-nested-within-trainee design. Score variation attributable to the trainee for each competency item assessed (variance component) were estimated by the minimum-norm quadratic unbiased estimator. Score variance was used to estimate the number needed for a reliability value of 0.80. The trainee score variance for each of 14 competency items varied between 2.3% for emergency skills to 35.6% for communication skills, with an average for all competency items of 20.3%; the "Overall rating" competency item trainee variance was 28.8%. These variance components translated into 169, 7, 17 and 28 assessments needed for a reliability of 0.80, respectively. Most variation in assessment scores was due to measurement error, ranging from 97.7% for emergency skills to 63.4% for communication skills. Similar results have been demonstrated in previously published studies. In summary, overall supervisors' workplace based assessments have poor reliability and are not suitable for use in certification processes in their current form. The marked variation in the supervisors' reliability in assessing different competencies indicates that supervisors may be able to assess some with acceptable reproducibility; in this case communication and possibly overall competence. However, any continued use of this format for assessment of trainee competencies necessitates the identification of what supervisors in different institutions can reliably assess rather than continuing to impose false expectations from unreliable assessments.

show abstract

Validity of three clinical performance assessments of internal medicine clerks

Cited by 56 publications

References 0 publications

The Anatomy Competence Score—A new marker for anatomical ability

The Anatomy Competence Score—A new marker for anatomical ability

The Objective Structured Clinical Examination

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

Contact Info

Product

Resources

About