Classifying Firearm Injury Intent in Electronic Hospital Records Using Natural Language Processing

Abstract

ImportanceInternational Classification of Diseases–coded hospital discharge data do not accurately reflect whether firearm injuries were caused by assault, unintentional injury, self-harm, legal intervention, or were of undetermined intent. Applying natural language processing (NLP) and machine learning (ML) techniques to electronic health record (EHR) narrative text could be associated with improved accuracy of firearm injury intent data.ObjectiveTo assess the accuracy with which an ML model identified firearm injury intent.Design, Setting, and ParticipantsA cross-sectional retrospective EHR review was conducted at 3 level I trauma centers, 2 from health care institutions in Boston, Massachusetts, and 1 from Seattle, Washington, between January 1, 2000, and December 31, 2019; data analysis was performed from January 18, 2021, to August 22, 2022. A total of 1915 incident cases of firearm injury in patients presenting to emergency departments at the model development institution and 769 from the external validation institution with a firearm injury code assigned according to International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Clinical Modification (ICD-10-CM), in discharge data were included.ExposuresClassification of firearm injury intent.Main Outcomes and MeasuresIntent classification accuracy by the NLP model was compared with ICD codes assigned by medical record coders in discharge data. The NLP model extracted intent-relevant features from narrative text that were then used by a gradient-boosting classifier to determine the intent of each firearm injury. Classification accuracy was evaluated against intent assigned by the research team. The model was further validated using an external data set.ResultsThe NLP model was evaluated in 381 patients presenting with firearm injury at the model development site (mean [SD] age, 39.2 [13.0] years; 348 [91.3%] men) and 304 patients at the external development site (mean [SD] age, 31.8 [14.8] years; 263 [86.5%] men). The model proved more accurate than medical record coders in assigning intent to firearm injuries at the model development site (accident F-score, 0.78 vs 0.40; assault F-score, 0.90 vs 0.78). The model maintained this improvement on an external validation set from a second institution (accident F-score, 0.64 vs 0.58; assault F-score, 0.88 vs 0.81). While the model showed some degradation between institutions, retraining the model using data from the second institution further improved performance on that site’s records (accident F-score, 0.75; assault F-score, 0.92).Conclusions and RelevanceThe findings of this study suggest that NLP ML can be used to improve the accuracy of firearm injury intent classification compared with ICD-coded discharge data, particularly for cases of accident and assault intents (the most prevalent and commonly misclassified intent types). Future research could refine this model using larger and more diverse data sets.

EMS injury cause codes more accurate than emergency department visit ICD-10-CM codes for firearm injury intent in North Carolina

“…• When the narrative described an incident as a drive-by shooting or shot by someone else while in a car with no documented/undetermined injury cause, intent was classified as assault. This approach is consistent with federal coding recommendations [14][15][16][17] and prior studies with manual firearm injury intent classifications.…”

“…For example, while some studies estimate unintentional firearm injury to represent 16% or less of all nonfatal firearm injury, approximately 60% of ED firearm injury visits in NC are identified as unintentional based on the ICD-10-CM external mechanism of injury codes [ 3 , 4 ]. Several other studies that have documented these coding intent discrepancies and the potential overclassification of nonfatal firearm injury as unintentional relied on urban, Level I Trauma center data from a small number of hospitals [ 5 , 6 ]. For comparison, we aimed to quantify any coding intent discrepancies using data from a statewide system in a predominantly rural state.…”

Using natural language processing in emergency medicine health service research: A systematic review and meta‐analysis

“…Overall, this meta-analysis included 27 studies with 179,109 patients and 9025 images. It consisted of four studies involving 23,707 patients focused on syndromic surveillance, 27,29,33,41 12 studies covering 155,402 patients targeting diseases/events/syndromes recognition, [36][37][38][39][40][42][43][44][45]47,48,53 and 11 studies involving 9025 images concentrating on radiology interpretations. 28,[30][31][32]34,35,46,[49][50][51][52] Detailed descriptions of each study are provided in Table 2.…”

“…Confusion matrices were unsuccessfully derived from two studies, 39,46 and the other 10 reported varying performance. 31,33,36,[39][40][41]43,[45][46][47][48]53 The sensitivity analysis revealed that the mean sensitivity was 0.82 (95% CI 0.73-0.88), specificity was 0.95 (95% CI 0.92-0.97), PPV was 0.14 (95% CI 0.14-0.14), NPV was 1.00 (95% CI 1.00-1.00), F I G U R E 1 PRISMA flow diagram: using NLP in EM research. NLP, natural language processing.…”