Can students’ reasons for choosing set answers to ethical vignettes be reliably rated? Development and testing of a method

Abstract

Although ethics is an important part of modern curricula, measures of students' ethical disposition have not been easy to develop. A potential method is to assess students' written justifications for selecting one option from a preset range of answers to vignettes and compare these justifications with predetermined 'expert' consensus. We describe the development of and reliability estimation for such a method -- the Ethics in Health Care Instrument (EHCI). Seven raters classified the responses of ten subjects to nine vignettes, on two occasions. The first stage of analysis involved raters' judging how consistent with consensus were subjects' justifications using generalizability theory, and then rating consensus responses on the action justification and values recognition hierarchies. The inter-rater reliability was 0.39 for the initial rating. Differential performance on questions was identified as the largest source of variance. Hence reliability was investigated also for students' total scores over the nine consensus vignettes. Rater effects were the largest source of variance identified. Examination of rater performance showed lack of rater consistency. D-studies were performed which showed acceptable reliability could nevertheless be obtained using four raters per EHCI. This study suggests that the EHCI has potential as an assessment instrument although further testing is required of all components of the methodology.

From behaviours to attributions: further concerns regarding the evaluation of professionalism

“…We have previously argued that we cannot judge behaviours in isolation; we must take the context into account, as well as how the student resolved the conflict 3 . However, as others have found, 20 presenting faculty attendings with more information about students’ rationales for action did not necessarily make it easier for them to judge the action and, of course, the added step of assigning a numerical score to the judgement represented an additional hurdle that greatly affected reliability. Thus, although it is possible that building a scoring template based on this most recent set of interview data (and incorporating elements related to insight, responsibility and putting the patient first) might lead to better scoring agreement, based on the research to date, we are generally less optimistic about the success of such a strategy.…”

Developing professionalism in Italian medical students: an educational framework

“…The use of vignettes to contextualize ethical dilemmas is a method of challenging students’decision-making ability and to measure their proposed behavior 12. The vignettes can be written or videotaped scenarios 13.…”

Students’ attitudes and potential behaviour to a competent patient’s request for withdrawal of treatment as they pass through a modern medical curriculum

“…These categories were derived from the reflections of the Glasgow researchers (JG, LS, JM) and one of the original developers of the instrument (Ken Kipnis, University of Hawaii), and grounded in responses given by students in both Hawaii and Glasgow. 12 Students' written categories were also compared with their choice of preset answer to help determine whether their thinking was consistent with professional consensus.…”

“…5 Comparison of the positions of students' justifications on the hierarchies before and after instruction was used as measures of change following instruction. 12 The reliability of the categorisation/classification process was estimated using the kappa coefficient (table 1). The…”