Supporting the classification of patients in public hospitals in Chile by designing, deploying and validating a system based on natural language processing

Abstract

Background In Chile, a patient needing a specialty consultation or surgery has to first be referred by a general practitioner, then placed on a waiting list. The Explicit Health Guarantees (GES in Spanish) ensures, by law, the maximum time to solve 85 health problems. Usually, a health professional manually verifies if each referral, written in natural language, corresponds or not to a GES-covered disease. An error in this classification is catastrophic for patients, as it puts them on a non-prioritized waiting list, characterized by prolonged waiting times. Methods To support the manual process, we developed and deployed a system that automatically classifies referrals as GES-covered or not using historical data. Our system is based on word embeddings specially trained for clinical text produced in Chile. We used a vector representation of the reason for referral and patient's age as features for training machine learning models using human-labeled historical data. We constructed a ground truth dataset combining classifications made by three healthcare experts, which was used to validate our results. Results The best performing model over ground truth reached an AUC score of 0.94, with a weighted F1-score of 0.85 (0.87 in precision and 0.86 in recall). During seven months of continuous and voluntary use, the system has amended 87 patient misclassifications. Conclusion This system is a result of a collaboration between technical and clinical experts, and the design of the classifier was custom-tailored for a hospital's clinical workflow, which encouraged the voluntary use of the platform. Our solution can be easily expanded across other hospitals since the registry is uniform in Chile.

Training and intrinsic evaluation of lightweight word embeddings for the clinical domain in Spanish

“…Our group has experience using static word embeddings for patient classification deployed in a hospital (Villena et al, 2021b ) and using stacked embeddings that combine both static and contextualized embeddings for named entity recognition (Báez et al, 2020 ; Báez et al, 2022 ) and automatic coding (Villena et al, 2021a ). Evaluating the automatic translation of clinical sentences has pointed us the need for creating reliable intrinsic tests created from scratch for the Spanish language, which can be valuable for both static and contextual word embeddings.…”

“…These referrals were obtained via Chilean Transparency Law, a government-wide initiative giving every citizen the right to request anonymized public data (Martinez et al, 2019 ). We got access to 11,826,843 referrals collected by the authors (Villena et al, 2021b ), and a subset of this corpus has been annotated with entities and relations clinically relevant 4 . These diagnostic suspicions were written directly by general practitioners in a primary-care setting.…”

“…For the computation of word embeddings a training corpus is needed. For this reason, we extracted data from three sources: (1) clinical narratives from The Chilean Waiting List Corpus, which is a collection of diagnostic suspicions from the waiting list in Chilean public hospitals (Báez et al, 2020 ; Báez et al, 2022 ; Villena et al, 2021b ), (2) a medical journal corpus extracted from the SciELO library, which is a collection of articles from several medical journals in Spanish (Villena et al, 2020 ) and (3) a corpus constructed from the Unified Medical Language System (UMLS) (Bodenreider, 2004 ) term graph. We computed embeddings using Word2vec and fastText algorithms and validated them using classic intrinsic evaluation tests adapted to Spanish, such as word pair similarity and semantic textual similarity.…”

Correction to: Supporting the classifcation of patients in public hospitals in Chile by designing, deploying and validating a system based on natural language processing

“…1186/ s12911-021-01565-z. *Correspondence: jdunstan@uchile.cl 1 Center for Mathematical Modeling -CNRS UMI2807, Faculty of Physical and Mathematical Sciences, University of Chile, Santiago, Chile Full list of author information is available at the end of the article…”

“…Following publication of the original article [ 1 ], it was reported that the list of authors was published in the incorrect order. Jocelyn Dunstan was erroneously listed as the first author instead of the last.…”