Faculty Evaluation by Residents in an Emergency Medicine Program: A New Evaluation Instrument

Abstract

Abstract. Objective: Evaluation of preceptors in training programs is essential; however, little research has been performed in the setting of the emergency department (ED). The goal of this pilot study was to determine the validity and reliability of a faculty evaluation instrument-the Emergency Rotation (ER) scale-developed specifically for use in emergency medicine (EM). Methods: A prospective study comparing the ER scale with two alternative faculty evaluation instruments was completed in three of the five EDs affiliated with an EM teaching program, where emergency physicians are members of the clinical teaching faculty. The participants were 18 residents (postgraduate years 1, 2, and 3) who were completing four-week clinical rotations in EM. Residents at the end of the rotation recorded their evaluations of each emergency physician with whom they had clinical encounters on the following evaluation tools: the ER scale, a longer validated scale (Irby), and a global assessment scale (GAS). Domain scores were correlated with the previously validated scale and the GAS to determine validity using a multitrait-multimethod matrix. The reliability of the ER scale was measured using a Chronbach's alpha coefficient. Results: Forty-eight preceptor evaluations were completed on 29 individual preceptors. The rating of preceptors was high using the ER scale (median: 16 of 20; IQR: 13, 18), Irby (median: 300 of 378; IQR: 267, 321), or GAS (mean: 7.8 of 10; SD: 1.3). Domain scores for each tool were used in the multitrait-multimethod matrix and the correlations between a previously validated tool and the ER scale were found to be high (>0.70) in the various domains. The internal consistency of the ER scale was also high (r = 0.85). Conclusions:The ER scale appears to be valid and reliable. It performs well when compared with previously psychometrically tested tools. It is a sensible, welladapted tool for the teaching environment offered by EM.

The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis

“…While most instruments had adequate internal consistency reliability, few studies reported generalizability coefficients or validity data beyond factor analysis. A notable exception was the Steiner et al (2000) study which employed a multitrait, multimethod approach to construct validity.…”

“…Many evaluation instruments have been developed to provide feedback to clinical teachers, department heads, and program directors. There has been a considerable effort devoted to investigate the psychometric properties of instruments to assess clinical teaching effectiveness (Ramsbottom-Lucier et al 1994;Solomon et al 1997;Bardes et al 1998;Litzelman et al 1998Litzelman et al , 1999Copeland & Hewson 1999;Steiner et al 2000;Williams et al 2001Williams et al , 2002Snell et al 2002;van der Hem-Stokroos et al 2003;Beckman et al 2004Beckman et al , 2005Leamon & Fields 2005;Bierer & Hull 2007).…”

Factor instability of clinical teaching assessment scores among general internists and cardiologists

“…7,8 Numerous studies have evaluated the psychometric characteristics of clinical teaching assessments. These studies included ratings of faculty by students, [8][9][10][11][12][13][14][15][16][17] residents 7,18-20 and peers 21,22 in diverse educational settings, such as internal medicine, 19,[21][22][23][24] family medicine, 14,18,25 emergency medicine, 26 gynaecology 13 and surgery. 10,27,28 However, our review of the literature 5 revealed that although factor analytic studies of clinical teaching assessments abound, [7][8][9]11,[13][14][15][16]19,23,27,28 none have described the factorial stability of teaching assessment scores across different medical specialties.…”

Developing a Standardized Faculty Evaluation in an Emergency Medicine Residency

“…Faculty evaluation enables a residency program to accomplish several goals. First, faculty evaluation provides an opportunity for assessment with the intent of reflection and ultimate improvement in teaching performance (4,5). Second, faculty evaluations can support promotion and tenure, particularly at institutions with teaching tracks.…”

“…The emergency department (ED) is a unique learning environment with unique challenges (4). The ED is a unique learning environment secondary to the chaotic nature of an unscheduled environment.…”