A predictive validity study of medical judgment vignettes to assess students’ noncognitive attributes: A 3-year prospective longitudinal study

Abstract

Background: The admissions interview still remains the most common approach used to describe candidates' noncognitive attributes for medical school. Aim: In this prospective study, we have investigated the predictive validity of a semi-structured interview for admissions to medical school based on medical judgment vignettes: (1) ethical decision-making (moral), (2) relationships with patients and their families (altruistic), and (3) roles and responsibilities in professional relationships (dutiful). Method: A group of 26 medical students from the Class of 2007 participated in the interview process and provided their subsequent performance results from clerkship 3 years later. Results: Inter-rater reliability of the scored interviews was high (k ¼ 0.96). Our results provided evidence for both convergent and divergent predictive validity. Medical judgment vignettes scores correlated significantly with seven mandatory clerkship rotation in-training evaluation reports (r ¼ 0.39, p < 0.05; to r ¼ 0.55, p < 0.01). Conclusion: This semi-structured interview based on clearly defined and scored medical judgment vignettes that focus on the assessment of medical students' noncognitive attributes is promising for student's selection into medical school. The high reliability and evidence of predictive validity of clinical performance over a 3-year period suggests a workable approach to the assessment of 'compelling personal characteristics' beyond merely cognitive variables.

A Generalizability Analysis of a Veterinary School Multiple Mini Interview: Effect of Number of Interviewers, Type of Interviewers, and Number of Stations

“…There is preliminary evidence of content and construct validity of this process in veterinary medicine, and Donnon et al demonstrated predictive validity of these vignettes compared to specific criterion meant to assess the same constructs. 2,10 The results of this report provide a foundation for further predictive validity studies. We are making an explicit effort to assess specific attributes in the interview process; the next step is to create appropriate measures throughout the program that reflect these same attributes to assess the validity of the process.…”

Dresden Faculty selection procedure for medical students: what impact does it have, what is the outcome?

“…We will therefore continue to analyze the different cohorts during the clinical years of the curriculum in order to validate our selection instruments by looking at students’ competencies during the clinical clerkships/electives [10]. Additionally, the Second National Medical Exam (SNME), which is taken after 6 years of education and which consists of a written and an oral part, will be used to analyze differences between the cohorts.…”

A preliminary analysis of a competence assessment tool for postgraduate training programmes in clinical and forensic psychology

“…The global subjective rating of a senior clinician following observation of clinical work is a common method of competence assessment, but it does not support the provision of systematic feedback on student development of specific competencies. Nonetheless, one clinical performance measure should correlate well with another (Campos‐Outcalt, Watkins, Fulginiti, Kutob, & Gordon, ; Donnon, Oddone‐Paolucci, & Violato, ). Therefore, to examine the convergent validity of the CSAT, a senior clinical psychologist was asked to provide a global judgement of student competence following viewing of Item 7 “Presents Formulation to Client.” This item was selected as it demonstrated a strong correlation with the total CSAT score of r = .83.…”