A fast, resource efficient, and reliable rule-based system for COVID-19 symptom identification

Abstract

Objective With COVID-19 there was a need for rapidly scalable annotation system that facilitated real-time integration with clinical decision support systems (CDS). Current annotation systems suffer from high resource utilization and poor scalability limiting real-world integration with CDS. A potential solution to mitigate these issues is to use the rule-based gazetteer developed at our institution. Materials and Methods Performance, resource utilization and runtime of the rule-based gazetteer was compared with five annotation systems: BioMedICUS, cTAKES, MetaMap, CLAMP and MedTagger. Results This rule-based gazetteer was fastest, had low resource footprint and similar performance for weighted micro-average and macro-average measures of precision, recall and f1-score compared to other annotation systems. Discussion Opportunities to increase its performance include fine-tuning lexical rules for symptom identification. Additionally, it could run on multiple compute nodes for faster runtime. Conclusion This rule-based gazetteer overcame key technical limitations facilitating real-time symptomatology identification for COVID-19 and integration of unstructured data elements into our CDS. It is ideal for large-scale deployment across a wide variety of health care settings for surveillance of acute COVID-19 symptoms for integration into prognostic modeling. Such a system is currently being leveraged for monitoring of post-acute sequelae of COVID-19 (PASC) progression in COVID-19 survivors. This study conducted the first in-depth analysis and developed a rule-based gazetteer for COVID-19 symptom extraction with the following key features: low processor and memory utilization, faster runtime and similar weighted micro-average and macro-average measures for precision, recall and f1-score compared to industry standard annotation systems. Lay Summary With COVID-19 came an unprecedented need to identify symptoms of COVID-19 patients under investigation (PUIs) in a time sensitive, resource-efficient and accurate manner. While available annotation systems perform well for smaller healthcare settings, they fail to scale in larger healthcare systems where 10,000+ clinical notes are generated a day. This study covers 3 improvements addressing key limitations of current annotation systems. (1) High resource utilization and poor scalability of existing annotation systems. The presented rule-based gazetteer is a high-throughput annotation system for processing high volume of notes, thus, providing opportunity for clinicians to make more informed time-sensitive decisions around patient care. (2) Equally important is our developed rule-based gazetteer performs similar or better than current annotation systems for symptom identification. (3) Due to minimal resource needs of the rule-based gazetteer, it could be deployed at healthcare sites lacking a robust infrastructure where industry standard annotation systems cannot be deployed because of low resource availability.

Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study

“…However, an AI-enabled clinical decision support system may add additional information, which ED providers can integrate into clinical decision-making when developing a differential diagnosis and determining whether confirmatory testing and isolation for COVID-19 are necessary. We propose the integration of structured and unstructured ( 23 ) data from electronic medical records into model training in the future.…”

Clinical decision support systems (CDSS) in assistance to COVID‐19 diagnosis: A scoping review on types and evaluation methods

“…These studies also recommended the use of CDSS in future research. Finally, 68 articles met all the inclusion criteria 5,17,34–99 . The flowchart of the selection process is shown in Figure 1.…”

“…The most frequently used method for this purpose was the ICDSS based on ML (nonknowledge‐based CDSS), followed by ICDSS based on ES (knowledge‐based CDSS). Most studies have indicated that using CDSS have positively impacted the accurate diagnosis of COVID‐19 5,17,34–62,64–84,86–90,92–99 …”

“…Some studies used a combination of ML methods to develop CDSS (n = 14) [34][35][36][37]39,[42][43][44]53,[55][56][57]70,84 (Appendix 2). Rule sets (n = 8), 5,17,88,89,93,94,96,97 fuzzy mathematical models (n = 5), 86,[90][91][92]95 and ontology (n = 3) 87,98,99 were used for developing ICDSS based on ES (knowledge-based CDSS), respectively (Appendix 2).…”

“…101 The reviewed studies extracted data from various sources, such as observed symptoms, specific medical measurements, pre-existing medical conditions or any hospitalization history, recent PCR test results, clinical guidelines, websites (e.g., WHO), and knowledge of experts to develop rule sets. 5,17,[86][87][88][89][90][91][92][93][94][95][96][97][98][99] Therefore, knowledge-based CDSS in these studies can provide a comprehensive view of the patient's health information. This feature enables health practitioners to access treatment recommendations and risk classification, recommend test lists, quickly track test results and symptoms, and access clinical guidelines in the COVID-19 pandemic.…”