Factor instability of clinical teaching assessment scores among general internists and cardiologists

Beckman, Thomas J.; Cook, David A.; Mandrekar, Jayawant N.

doi:10.1111/j.1365-2929.2006.02632.x

Cited by 37 publications

(36 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our conceptual framework was based on previous factor analytic studies demonstrating that assessments of clinical teachers by resident physicians reduce to interpersonal, clinical-teaching, and efficiency domains (Beckman and Mandrekar 2005;Beckman et al 2006). These studies, in turn, built upon previous research showing that clinical performance assessments of learners and clinical teaching assessments, which were originally believed to tap multiple dimensions, measured the broader attributes of humanism and clinical skills (McLeod et al 1993;Durning et al 2003;Silber et al 2004).…”

Section: Conceptual Framework and Qualitative Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Resident physicians’ perspectives on effective outpatient teaching: a qualitative study

Kisiel

Bundrick

Beckman

2009

Adv in Health Sci Educ

View full text Add to dashboard Cite

Learning theories, which suggest that experienced faculty use collaborative teaching styles, are reflected in qualitative studies of learners in hospital settings. However, little research has used resident focus groups to explore characteristics of successful teachers in outpatient clinics. Therefore, focus group discussions with first through third-year internal medicine residents at a large academic medical center were conducted to better understand residents' perspectives on effective outpatient teaching. A group facilitator solicited residents' reflections, based on their lived experiences, on teaching domains from previous factor analytic studies: interpersonal, clinical-teaching, and efficiency. Researchers coded focus group transcripts and identified themes within the domains. Final themes were determined by consensus. Leading themes were "kindness" and "teacher-learner relationships." Junior residents were sensitive to faculty who were brusque, harsh, and degrading. Senior residents respected faculty who were humble, collaborative, and allowed residents to co-manage teaching encounters. Seniors emphasized the importance of faculty role-modelling and preferentially staffed with experts to "gain wisdom from experience." Overall, residents expressed that effective learning requires grounded teacher-learner relationships. These findings support learning theories and previous factor analytic studies. However, this qualitative study provided insights that could not be gleaned from assessment scores alone.

show abstract

Section: Conceptual Framework and Qualitative Methodsmentioning

confidence: 99%

“…We previously demonstrated that teaching assessments at our institution reduce to interpersonal, clinical-teaching, and efficiency domains (Beckman and Mandrekar 2005;Beckman et al 2006). Nonetheless, quantitative assessment studies often provide a limited perspective.…”

Section: Introductionmentioning

confidence: 99%

Resident physicians’ perspectives on effective outpatient teaching: a qualitative study

Kisiel

Bundrick

Beckman

2009

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…In contrast, the outcomes in our study consisted of learner evaluations that were supported by validity evidence, 33 including content, internal structure, and relationships to other variables, such as faculty characteristics and resident well-being. 30,31,34 We demonstrated that low-performing teachers benefited from faculty development more than their topperforming colleagues. Our findings extend the results of a retrospective study of surgeons by Cohen et al, 35 which revealed that good and average teachers had temporally stable scores, but that poor teachers tended to progress.…”

Section: Discussionmentioning

confidence: 92%

Success of a Faculty Development Program for Teachers at the Mayo Clinic

Lee¹,

Lee²,

Reed³

et al. 2014

Journal of Graduate Medical Education

View full text Add to dashboard Cite

Background There has been limited research on the improvement of underperforming clinical teachers. Objective To determine whether a faculty development program could improve the evaluations of clinical teachers in an internal medicine residency program. Methods A total of 123 teachers completed faculty development at the Mayo Clinic from 2009 to 2012. The faculty enhancement and education development program (FEED) consists of 6 interactive, small group, 2-hour sessions taught by experienced Mayo Clinic faculty over 1 year. These sessions address the following competencies: asking questions, diagnosing learners, giving feedback, using teaching frameworks, recognizing learning styles, and providing clinical supervision. Resident-of-faculty Mayo teaching effectiveness (MTE) scores have previously demonstrated content, internal structure, and criterion validity. Teachers were grouped into the top 80% or the bottom 20%, according to baseline MTE scores. Mixed linear models were used to compare these groups regarding changes in MTE scores after completion of FEED. Results were adjusted for teacher age, sex, medical specialty, academic rank, and teaching awards. Results For all participants combined, the adjusted MTE scores (mean; standard error) improved from baseline (3.80; 0.04) to completion of FEED (3.93; 0.04; P < .001). However, the bottom 20% had a significantly greater improvement in scores than the top 80% (score-change difference = 0.166, P < .001). Conclusions We describe a low-intensity faculty development intervention that benefited all clinical teachers, but was particularly effective for underperforming teachers in internal medicine. The approach may be suitable for adoption or adaptation in other graduate medical education programs.

show abstract

“…The reliability of similar types of assessment processes have been variable, although many have claimed adequate reliability (Keck and Arnold 1979;Kwolek et al 1997;Magzoub et al 1998;Kreiter et al 1998;Nasca et al 2002;Durning et al 2005;Beckman et al 2006;Cohen et al 2009;Kreiter et al 1998), others have been either equivocal (Cowles and Kubany 1959;Hull et al 1995;Schwanz et al 1995;Williams et al 2004), or found the reliability not acceptable (Levine and McGuire 1971;Davis et al 1986;Thompson et al 1990;Metheny 1991;Ryan et al 1996;Pulito et al 2007;Searle 2008). A common problem with many of the studies claiming reliability for this form of competency assessment has been the inappropriate use of the alpha coefficient for nested and/or unbalanced designs which appears to be common for workplace-based assessments (Keck and Arnold 1979;Magzoub et al 1998;Nasca et al 2002;Durning et al 2005;Cohen et al 2009).…”

Section: Comparison To Other Studiesmentioning

confidence: 93%

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

McGill

Vleuten

Clarke

2011

Adv in Health Sci Educ

View full text Add to dashboard Cite

Even though rater-based judgements of clinical competence are widely used, they are context sensitive and vary between individuals and institutions. To deal adequately with rater-judgement unreliability, evaluating the reliability of workplace rater-based assessments in the local context is essential. Using such an approach, the primary intention of this study was to identify the trainee score variation around supervisor ratings, identify sampling number needs of workplace assessments for certification of competence and position the findings within the known literature. This reliability study of workplace-based supervisors' assessments of trainees has a rater-nested-within-trainee design. Score variation attributable to the trainee for each competency item assessed (variance component) were estimated by the minimum-norm quadratic unbiased estimator. Score variance was used to estimate the number needed for a reliability value of 0.80. The trainee score variance for each of 14 competency items varied between 2.3% for emergency skills to 35.6% for communication skills, with an average for all competency items of 20.3%; the "Overall rating" competency item trainee variance was 28.8%. These variance components translated into 169, 7, 17 and 28 assessments needed for a reliability of 0.80, respectively. Most variation in assessment scores was due to measurement error, ranging from 97.7% for emergency skills to 63.4% for communication skills. Similar results have been demonstrated in previously published studies. In summary, overall supervisors' workplace based assessments have poor reliability and are not suitable for use in certification processes in their current form. The marked variation in the supervisors' reliability in assessing different competencies indicates that supervisors may be able to assess some with acceptable reproducibility; in this case communication and possibly overall competence. However, any continued use of this format for assessment of trainee competencies necessitates the identification of what supervisors in different institutions can reliably assess rather than continuing to impose false expectations from unreliable assessments.

show abstract

Factor instability of clinical teaching assessment scores among general internists and cardiologists

Cited by 37 publications

References 31 publications

Resident physicians’ perspectives on effective outpatient teaching: a qualitative study

Resident physicians’ perspectives on effective outpatient teaching: a qualitative study

Success of a Faculty Development Program for Teachers at the Mayo Clinic

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review

Contact Info

Product

Resources

About