Assessing the assessments: U.K. dermatology trainees’ views of the workplace assessment tools

Cohen, S. N.; Farrant, P.; Taibjee, Saleem M.

doi:10.1111/j.1365-2133.2009.09097.x

Cited by 49 publications

(48 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This is consistent with previous findings for video-assisted assessment and the use of DOPS in daily practice [24][25][26][27]. Qualitative investigations using stakeholder feedback have produced mixed results of DOPS feasibility and acceptability [26][27][28][29]. The DOPS assessment required 6.5 min (on average), potentially supporting its feasibility in practice.…”

Section: Discussionsupporting

confidence: 88%

Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound‐guided regional anaesthesia

et al. 2014

View full text Add to dashboard Cite

SummaryAssessment tools must be investigated for reliability, validity and feasibility before being implemented. In 2013, the Australian and New Zealand College of Anaesthetists introduced workplace-based assessments, including a direct observation of a procedural skills assessment tool. The objective of this study was to evaluate the psychometric properties of this assessment tool for ultrasound-guided regional anaesthesia. Six experts assessed 30 video-recorded trainee performances of ultrasound-guided regional anaesthesia. Inter-rater reliability, assessed using absolute agreement intraclass correlation coefficients, varied from 0.10 to 0.49 for the nine individual nine-point scale items, and was 0.25 for a 'total score' of all items. Internal consistency was measured by correlation between 'total score' and 'overall performance' scale item (r = 0.68, p < 0.001). Construct validity was demonstrated by the 'total score' correlating with trainee experience (r = 0.51, p = 0.004). The mean time taken to complete assessments was 6 min 35 s.

show abstract

Section: Discussionsupporting

confidence: 88%

Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound‐guided regional anaesthesia

et al. 2014

View full text Add to dashboard Cite

show abstract

“…The study purposefully recruited in sites with welldeveloped appraisal systems to enable participants to reflect on using multisource feedback in appraisal and revalidation, rather than focusing on problems with their appraisal process. Although recent international research has explored the relationship between multisource feedback, appraisal, and changing practice, 11,13,30,31 this exploratory study provides new insight into how the emergent plans for the revalidation of UK doctors might influence the relationship between multisource feedback and appraisal.…”

Section: Strengths and Limitationsmentioning

confidence: 99%

Multisource feedback questionnaires in appraisal and for revalidation: a qualitative study in UK general practice

et al. 2012

View full text Add to dashboard Cite

show abstract

“…The reliability of similar types of assessment processes have been variable, although many have claimed adequate reliability (Keck and Arnold 1979;Kwolek et al 1997;Magzoub et al 1998;Kreiter et al 1998;Nasca et al 2002;Durning et al 2005;Beckman et al 2006;Cohen et al 2009;Kreiter et al 1998), others have been either equivocal (Cowles and Kubany 1959;Hull et al 1995;Schwanz et al 1995;Williams et al 2004), or found the reliability not acceptable (Levine and McGuire 1971;Davis et al 1986;Thompson et al 1990;Metheny 1991;Ryan et al 1996;Pulito et al 2007;Searle 2008). A common problem with many of the studies claiming reliability for this form of competency assessment has been the inappropriate use of the alpha coefficient for nested and/or unbalanced designs which appears to be common for workplace-based assessments (Keck and Arnold 1979;Magzoub et al 1998;Nasca et al 2002;Durning et al 2005;Cohen et al 2009).…”

Section: Comparison To Other Studiesmentioning

confidence: 99%

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

McGill

Vleuten

Clarke

2011

Adv in Health Sci Educ

View full text Add to dashboard Cite

Even though rater-based judgements of clinical competence are widely used, they are context sensitive and vary between individuals and institutions. To deal adequately with rater-judgement unreliability, evaluating the reliability of workplace rater-based assessments in the local context is essential. Using such an approach, the primary intention of this study was to identify the trainee score variation around supervisor ratings, identify sampling number needs of workplace assessments for certification of competence and position the findings within the known literature. This reliability study of workplace-based supervisors' assessments of trainees has a rater-nested-within-trainee design. Score variation attributable to the trainee for each competency item assessed (variance component) were estimated by the minimum-norm quadratic unbiased estimator. Score variance was used to estimate the number needed for a reliability value of 0.80. The trainee score variance for each of 14 competency items varied between 2.3% for emergency skills to 35.6% for communication skills, with an average for all competency items of 20.3%; the "Overall rating" competency item trainee variance was 28.8%. These variance components translated into 169, 7, 17 and 28 assessments needed for a reliability of 0.80, respectively. Most variation in assessment scores was due to measurement error, ranging from 97.7% for emergency skills to 63.4% for communication skills. Similar results have been demonstrated in previously published studies. In summary, overall supervisors' workplace based assessments have poor reliability and are not suitable for use in certification processes in their current form. The marked variation in the supervisors' reliability in assessing different competencies indicates that supervisors may be able to assess some with acceptable reproducibility; in this case communication and possibly overall competence. However, any continued use of this format for assessment of trainee competencies necessitates the identification of what supervisors in different institutions can reliably assess rather than continuing to impose false expectations from unreliable assessments.

show abstract

Assessing the assessments: U.K. dermatology trainees’ views of the workplace assessment tools

Cited by 49 publications

References 11 publications

Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound‐guided regional anaesthesia

Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound‐guided regional anaesthesia

Multisource feedback questionnaires in appraisal and for revalidation: a qualitative study in UK general practice

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

Contact Info

Product

Resources

About