Accuracy of ‘My Gut Feeling:’ Comparing System 1 to System 2 Decision-Making for Acuity Prediction, Disposition and Diagnosis in an Academic Emergency Department

Abstract

IntroductionCurrent cognitive sciences describe decision-making using the dual-process theory, where a System 1 is intuitive and a System 2 decision is hypothetico-deductive. We aim to compare the performance of these systems in determining patient acuity, disposition and diagnosis.MethodsProspective observational study of emergency physicians assessing patients in the emergency department of an academic center. Physicians were provided the patient’s chief complaint and vital signs and allowed to observe the patient briefly. They were then asked to predict acuity, final disposition (home, intensive care unit (ICU), non-ICU bed) and diagnosis. A patient was classified as sick by the investigators using previously published objective criteria.ResultsWe obtained 662 observations from 289 patients. For acuity, the observers had a sensitivity of 73.9% (95% CI [67.7–79.5%]), specificity 83.3% (95% CI [79.5–86.7%]), positive predictive value 70.3% (95% CI [64.1–75.9%]) and negative predictive value 85.7% (95% CI [82.0–88.9%]). For final disposition, the observers made a correct prediction in 80.8% (95% CI [76.1–85.0%]) of the cases. For ICU admission, emergency physicians had a sensitivity of 33.9% (95% CI [22.1–47.4%]) and a specificity of 96.9% (95% CI [94.0–98.7%]). The correct diagnosis was made 54% of the time with the limited data available.ConclusionSystem 1 decision-making based on limited information had a sensitivity close to 80% for acuity and disposition prediction, but the performance was lower for predicting ICU admission and diagnosis. System 1 decision-making appears insufficient for final decisions in these domains but likely provides a cognitive framework for System 2 decision-making.

Eyeballing: the use of visual appearance to diagnose ‘sick’

“…Our hypothesis is that experts can diagnose ‘sick’ as claimed in the literature via heavy reliance on System 1 processing, as measured by short decision times and self‐report of pattern recognition . We suspect that accuracy at diagnosing ‘sick’ based on visual assessment alone will surpass accuracy at identifying patient disposition as other information from the history and physical examination, past medical history, social context and diagnostic test information is likely to factor into the admission decision …”

“…It helps clinicians anticipate the disposition (and required resources) for a patient: who can go home, and who needs to stay in hospital. In brief, the eyeball test is how clinicians decide who is sick and who is not …”

“…Whether or not experts can truly judge illness severity based on visual appearance alone is not clear. Prior studies showed that experts could accurately identify sick patients and predict disposition decisions . However, these studies provided much more context.…”

The inter-rater reliability of the Risk Instrument for Screening in the Community

“…One current understanding of decision-making processes called the dual-process theory describes two systems whereby we make decisions: the intuitive System 1 and the hypothetico-deductive System 2. Evidence suggests that the accuracy of System 1, 'gut feeling', applied by nurses in clinical practice, compares well (sensitivity of 80%) with the deliberate methodological System 2 approach [Cabrera et al, 2015]. Likewise, while it is often considered that risk tolerance increases with knowledge and experience, there is little evidence to support this in clinical practice [Considine].…”

Have we forgotten to teach how to think?

“…It is surprising how little we concentrate on the process of making a diagnosis during training. Different diagnostic approaches have been identified3 between junior doctors and we are increasingly realising how gut feeling impacts on decision-making 4 5. How guidelines might interact with experience and gut feeling has yet to be completely delineated but clearly in some cases guidelines trump gut feeling and vice versa.…”