Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study

Abstract

Background: Systematic assessment of clinical reasoning skills of medical students in clinical practice is very difficult. This is partly caused by the lack of understanding of the fundamental mechanisms underlying the process of clinical reasoning. Methods: We previously developed an observation tool to assess the clinical reasoning skills of medical students during clinical practice. This observation tool consists of an 11-item observation rating form (ORT). In the present study we verified the validity, reliability and feasibility of this tool and of an already existing post-encounter rating tool (PERT) in clinical practice among medical students during the internal medicine clerkship. Results: Six raters each assessed the same 15 student-patient encounters. The internal consistency (Cronbach's alfa) for the (ORT) was 0.87 (0.71-0.84) and the 5-item (PERT) was 0.81 (0.71-0.87). The intraclass-correlation coefficient for single measurements was poor for both the ORT; 0.32 (p < 0.001) as well as the PERT; 0.36 (p < 0.001). The Generalizability study (G-study) and decision study (D-study) showed that 6 raters are required to achieve a Gcoefficient of > 0.7 for the ORT and 7 raters for the PERT. The largest sources of variance are the interaction between raters and students. There was a consistent correlation between the ORT and PERT of 0.53 (p = 0.04). Conclusions: The ORT and PERT are both feasible, valid and reliable instruments to assess students' clinical reasoning skills in clinical practice.

Assessing clinical reasoning ability in fourth-year medical students via an integrative group history-taking with an individual reasoning activity

“…We created a group history taking with individual reasoning activity at Mackay Medical College which is an elective pre-clerkship education course for fourth-year medical students, with the aim of training and assessing students’ clinical reasoning ability. To evaluate clinical reasoning ability, Haring et al developed an observation tool that consists of an 11-item observation rating form and a post encounter rating tool which are both feasible, valid, and reliable to assess students’ clinical reasoning skills in clinical practice [ 11 ]. They also reported that by observing and assessing clinical reasoning during medical students’ history taking, general and specific phenomena could be used as indicators, including taking control, recognizing, and responding to relevant information, specifying symptoms, asking specific questions that point to pathophysiological thinking, placing questions in a logical order, checking agreement with patients, summarizing, and body language [ 12 ].…”

Feasibility and Effectiveness of Mini-Clinical Evaluation Exercise (Mini-CEX) in an Undergraduate Medical Program: A Study From Pakistan

“…These were assessed on a 3-point Likert scale (Needs Improvement, Improving and Satisfactory or Above), with justification required for each assessment and written feedback to improve that skill (Figure 1); effectiveness was assessed via a change in student performance between the first and second Mini-CEX. We also obtained feedback from students and faculty at the end of the study through specially designed and internally validated forms, and assessed the feasibility of the tool qualitatively via comments from students and faculty about ease and time of completion in real-time [10].…”

Assessing Clinical Reasoning Ability in Fourth-year Medical Students via an Integrative Group History-taking Workshop With an Individual Reasoning Teaching Programme

“…To evaluate clinical reasoning ability, Haring et al developed an observation tool that consists of an 11-item observation rating form and a post encounter rating tool which are both feasible, valid, and reliable to assess students' clinical reasoning skills in clinical practice. 10 They also reported that by observing and assessing clinical reasoning during medical students' history taking, general and speci c phenomena could be used as indicators, including taking control, recognising, and responding to relevant information, specifying symptoms, asking speci c questions that point to pathophysiological thinking, placing questions in a logical order, checking agreement with patients, summarising, and body language. 11 We modi ed the methods that were used for the GOSCE workshop to include history taking and clinical reasoning each time we let four to ve students visit a standardised patient with a clinical case to collect enough data, and then individually write down key information and make a correct differential diagnosis.…”