Development, Validation, and Implementation of a Medical Judgment Metric

Abstract

Background: Medical decision making is a critical, yet understudied, aspect of medical education. Aims: To develop the Medical Judgment Metric (MJM), a numerical rubric to quantify good decisions in practice in simulated environments; and to obtain initial preliminary evidence of reliability and validity of the tool. Methods: The individual MJM items, domains, and sections of the MJM were built based on existing standardized frameworks. Content validity was determined by a convenient sample of eight experts. The MJM instrument was pilot tested in four medical simulations with a team of three medical raters assessing 40 participants with four levels of medical experience and skill. Results: Raters were highly consistent in their MJM scores in each scenario (intraclass correlation coefficient 0.965 to 0.987) as well as their evaluation of the expected patient outcome (Fleiss’s Kappa 0.791 to 0.906). For each simulation scenario, average rater cut-scores significantly predicted expected loss of life or stabilization (Cohen’s Kappa 0.851 to 0.880). Discussion: The MJM demonstrated preliminary evidence of reliability and validity.

Medical Decision-Making in the Physician Hierarchy: A Pilot Pedagogical Evaluation

“…The Summa Health Institutional Review Board and NASA Johnson Space Center Committee for the Protection of Human Subjects were responsible for approving the study protocol. Detailed procedures of recruitment and rationale for participant selection criteria were reported in previous publications by Ahmed et al 22 and McCarroll et al 23 Overall, participants were recruited by fliers, emails, and department meetings at local hospitals and universities in Northeast Ohio during a 6-month timeframe. Participant recruitment focused on strict screening criteria to ensure adequate sampling across 3 physician groups to represent various levels of pedagogical transitions: administrative physicians (APs; n = 10), mastery level physicians (MPs; n = 10), and resident physicians (RPs; n = 10).…”

“…Detailed procedures of each simulation were reported in previous publications by Ahmed et al 22 and McCarroll et al 23 In summary, each group of participants (APs, MPs, and RPs) were scheduled 1 practice simulated scenario and 4 tested scenarios: Abdominal pain scenarios were biliary colic (BC) and renal colic (RC) simulations, whereas chest pain scenarios were cardiac ischemia with ST-segment elevation myocardial infarction (STEMI) and tension pneumothorax (PTX). Each participant was assigned to complete the scenarios in a random order decided by random numbers generator software in Microsoft Excel V.2007 (Redmond, Washington, USA).…”

“…The need for a consistent assessment tool led to the development of the Medical Judgment Metric (MJM). 22 The metric creates a standardized measurement of the complex multidimensional thinking required of strong clinicians. By design, the MJM measures decision-making in challenging environments and differentiates between physicians who can perform well on the earlier stated training checkpoints from those who excel at the most difficult part of their responsibilities, making decisions for patients in critical situations.…”

Medical judgement analogue studies with applications to spaceflight crew medical officer

“…The MJM is a tool that measures medical judgement in four clinical domains: history and physical, diagnostic, interpretation, and management, with a maximum score of 4 in each domain on a 0.5 interval scale up to an overall maximum score of 16. 25 …”