Advancements in extracting social determinants of health information from narrative text

Lybarger, Kevin; Walk, Oliver J. Bear Don’t; Yetisgen, Meliha; Uzuner, Özlem

doi:10.1093/jamia/ocad121

Cited by 4 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 Recent work has focused on developing natural language processing (NLP) and machine learning approaches to extract or infer SBDH from clinical narratives. 8,9 NLP approaches are rule-based and identify SBDH lexicons (keywords and/or phrases) using keyword matching or regular expressions. Identification of SBDH lexicons and NLP rules require considerable manual refinement.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, these approaches still require a considerable amount of manual effort for fine-tuning and may not be applicable to SBDH factors with low prevalence. 9 Recent studies explored augmentation of low prevalence SBDH using simulated synthetic data and showed that fine-tuned Flan-T5 models outperformed zero-shot Generative Pretrained Transformer (GPT) models. 16 In this study, using the publicly available MIMIC-III dataset, 17 we analyzed all clinical notes for over 46,000 patients to identify 15 different SBDH categories using a well-known mathematical approach, called Latent Semantic Indexing (LSI).…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Large-scale identification of social and behavioral determinants of health from clinical notes: Comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models

Roy,

Morrell,

Zhao

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: Social and behavioral determinants of health (SBDH) are associated with a variety of health and utilization outcomes, yet these factors are not routinely documented in the structured fields of electronic health records (EHR). The objective of this study was to evaluate different machine learning approaches for detection of SBDH from the unstructured clinical notes in the EHR. Methods: Latent Semantic Indexing (LSI) was applied to 2,083,180 clinical notes corresponding to 46,146 patients in the MIMIC-III dataset. Using LSI, patients were ranked based on conceptual relevance to a set of keywords (lexicons) pertaining to 15 different SBDH categories. For Generative Pretrained Transformer (GPT) models, API requests were made with a Python script to connect to the OpenAI services in Azure, using gpt-3.5-turbo-1106 and gpt-4-1106-preview models. Prediction of SBDH categories were performed using logistic regression model that included age, gender race and SBDH ICD-9 codes with a natural cubic spline of 2 degrees of freedom for age. Results: LSI retrieved patients according to 15 SBDH domains, with an overall average PPV ≥ 83%. Using manually curated gold standard (GS) sets for nine SBDH categories, the macro-F1 score of LSI (0.74) was better than ICD-9 (0.71) and GPT-3.5 (0.54), but lower than GPT-4 (0.80). Due to document size limitations, only a subset of the GS cases could be processed by GPT-3.5 (55.8%) and GPT-4 (94.2%), compared to LSI (100%). Using common GS subsets for nine different SBDH categories, the macro-F1 of ICD-9 combined with either LSI (mean 0.88, 95% CI 0.82-0.93), GPT-3.5 (0.86, 0.82-0.91) or GPT-4 (0.88, 0.83-0.94) was not significantly different. After including age, gender, race and ICD-9 in a logistic regression model, the AUC for prediction of six out of the nine SBDH categories was higher for LSI compared to GPT-4.0. Conclusions: These results demonstrate that the LSI approach performs comparable to more recent large language models, such as GPT-3.5 and GPT-4.0, when using the same set of documents. Importantly, LSI is robust, deterministic, and does not have document-size limitations or cost implications, which make it more amenable to real-world applications in health systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Large-scale identification of social and behavioral determinants of health from clinical notes: Comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models

Roy,

Morrell,

Zhao

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

JAMIA at 30: looking back and forward

Stead,

Miller,

Ohno-Machado

et al. 2023

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Model-based estimation of individual-level social determinants of health and its applications in All of Us

Kim,

Anthopolos,

et al. 2024

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objectives We introduce a widely applicable model-based approach for estimating individual-level Social Determinants of Health (SDoH) and evaluate its effectiveness using the All of Us Research Program. Materials and Methods Our approach utilizes aggregated SDoH datasets to estimate individual-level SDoH, demonstrated with examples of no high school diploma (NOHSDP) and no health insurance (UNINSUR) variables. Models are estimated using American Community Survey data and applied to derive individual-level estimates for All of Us participants. We assess concordance between model-based SDoH estimates and self-reported SDoHs in All of Us and examine associations with undiagnosed hypertension and diabetes. Results Compared to self-reported SDoHs, the area under the curve for NOHSDP is 0.727 (95% CI, 0.724-0.730) and for UNINSUR is 0.730 (95% CI, 0.727-0.733) among the 329 074 All of Us participants, both significantly higher than aggregated SDoHs. The association between model-based NOHSDP and undiagnosed hypertension is concordant with those estimated using self-reported NOHSDP, with a correlation coefficient of 0.649. Similarly, the association between model-based NOHSDP and undiagnosed diabetes is concordant with those estimated using self-reported NOHSDP, with a correlation coefficient of 0.900. Discussion and Conclusion The model-based SDoH estimation method offers a scalable and easily standardized approach for estimating individual-level SDoHs. Using the All of Us dataset, we demonstrate reasonable concordance between model-based SDoH estimates and self-reported SDoHs, along with consistent associations with health outcomes. Our findings also underscore the critical role of geographic contexts in SDoH estimation and in evaluating the association between SDoHs and health outcomes.

show abstract

Advancements in extracting social determinants of health information from narrative text

Cited by 4 publications

References 16 publications

Large-scale identification of social and behavioral determinants of health from clinical notes: Comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models

Large-scale identification of social and behavioral determinants of health from clinical notes: Comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models

JAMIA at 30: looking back and forward

Model-based estimation of individual-level social determinants of health and its applications in All of Us

Contact Info

Product

Resources

About