Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context

Abstract

Aim: Many instruments for evaluating clinical teaching have been developed, albeit most in Western countries. This study aims to develop a validated cultural and local context sensitive instrument for clinical teachers in an East Asian setting (Japan), Japanese Clinical Teacher Evaluation Sheet (JaCTES). Methods: A multicenter, cross-sectional evaluation study was conducted. We collected a total of 1368 questionnaires on 304 clinical teachers, completed by residents in 16 teaching hospitals. The construct validity was examined by conducting a factor analysis and using structural equation modeling (SEM). We also assessed the reliability using generalizability analysis and decision study. Results: Exploratory factor analysis resulted in three-factor (role model, teaching activities, and accessibility) model including 18 items. Confirmatory factor analysis was performed, using SEM. The comparative fit index was 0.931 and the root mean square error of approximation was 0.087, meaning an acceptable goodness of fit for this model. To obtain a reliable dependability-coefficient of at least 0.70 or higher, 5-8 resident responses are necessary. Discussion and conclusion: JaCTES is the first reported instrument with validity evidence of content and internal structure and high feasibility in Japan, an East Asian setting. Medical educators should be aware of the local context and cultural aspects in evaluating clinical teachers.

Assessment of physicians’ professional performance

“…The remaining tools most often neglected evidence for intended implications. Seven teaching performance tools collected evidence on all components of the argument 74,78,[83][84][85]91,92,96,98,99,101,103,106,108,109,111,113,115,117,118,[120][121][122][123]128,[131][132][133][134] . Thus, in total, only 12 (23%) of all 53 tools gathered evidence on all four components of Kane's validity argument.…”

“…For the teaching tools, the scoring inference for item development seems to be overlooked by most authors. Studies of twenty-one tools do not or only poorly disclose how tools were developed regarding the items, scoring, or scales 64,67,68,[74][75][76]78,82,84,87,[90][91][92]97,98,100,104,110,111,114,115,125,127,130 . Studies on the remaining 17 tools disclosed how items were developed based on a theoretical framework, peerreviewed literature, other documents, other validated tools, or expert opinions 65,66,69,72, Chain number 73,77,79,81,83,85,88,89,[93][94][95][96]99,[101][102][103][105][106][107][108][109]112,113,[116][117][118][119][120][121][122]…”

“…[19][20][21]23,27,56,63 For teaching performance tools, over a third of the tools (n = 28) did not mention whether raters could select "unable to assess." [64][65][66]69,70,[74][75][76][77][78][79][80][81][82][83][84][85][86][87][89][90][91][92][93][94][95][97][98][99][100]102,104,111,[114][115][116]119,121,125,127,129,130,134 . Scores and scales.…”

“…For teaching tools, seven studies investigated whether scores changed over time and showed an improvement in scores after one or several assessment periods 65,70,84,98,115,121,133 . One study found a significant change in scores after physicians received teacher training, and one study showed that after receiving the assessment feedback, faculty received significantly higher ratings over time 70,121 .…”

“…Therefore, the findings from these studies cannot readily be generalized to the larger population of physicians outside this Dutch context, as we are inevitably limited by our Western cultural context. Validity is thought to be culturally sensitive; it is in itself also determined by the cultural context in which we operate 84 . However, in the systematic review in Chapter 2, the validity evidence in the scientific literature was considered throughout the world, albeit limited to English texts.…”