Artificial Intelligence in Action: Addressing the COVID-19 Pandemic with Natural Language Processing

Chen, Qingyu; Leaman, Robert; Allot, Alexis; Luo, Ling; Wei, Chih-Hsuan; Yan, Shankai; Lu, Zhiyong

doi:10.1146/annurev-biodatasci-021821-061045

Cited by 50 publications

(36 citation statements)

References 113 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…NIH COVID-19 Portfolio, 19 CORD-19 [8] and LitCOVID [7] are among the initial efforts for COVID-19 datasets, and we find many other datasets since been, including, Covidex [66] and others, as listed in recent literature [16] , [24] . However, according to our preliminary empirical research, there is a significant overlap of articles in these datasets, which is understandable given that the COVID-19 articles are available in each of these datasets or scholarly repositories.…”

Section: Discussion: Limitations and Future Perspectivesmentioning

confidence: 99%

“…The CDC database also contains a collection of white papers and technical reports, which are also found in CORD-19. Other interfaces provide access to COVID-19 papers, such as iSearch, 6 Covidex, 7 SciSight 8 and others as mentioned in a review article [24] , these interfaces also rely on the CORD-19 initiative as their primary source of information.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A COVID-19 Search Engine (CO-SE) with Transformer-based architecture

Raza¹

2022

Healthcare Analytics

View full text Add to dashboard Cite

Section: Discussion: Limitations and Future Perspectivesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

A COVID-19 Search Engine (CO-SE) with Transformer-based architecture

Raza¹

2022

Healthcare Analytics

View full text Add to dashboard Cite

“…It is estimated that more than 23k published papers have been indexed on Web of Science and Scopus just between January 1 and June 30, 2020 [46]. In the NLP domain, researchers are focusing on processing pandemicgenerated data and working on various tasks including text classification, information retrieval, named entity recognition and knowledge discovery [32,127,242]. While there are many relevant papers, we list here some typical works that perform NLP techniques on COVID-19 data.…”

Section: Applications In Public Healthmentioning

confidence: 99%

Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review

Li¹,

Pan²,

Goldwasser³

et al. 2021

Preprint

View full text Add to dashboard Cite

Electronic health records (EHRs), digital collections of patient healthcare events and observations, are ubiquitous in medicine and critical to healthcare delivery, operations, and research. Despite this central role, EHRs are notoriously difficult to process automatically. Well over half of the information stored within EHRs is in the form of unstructured text (e.g. provider notes, operation reports) and remains largely untapped for secondary use. Recently, however, newer neural network and deep learning approaches to Natural Language Processing (NLP) have made considerable advances, outperforming traditional statistical and rule-based systems on a variety of tasks. In this survey paper, we summarize current neural NLP methods for EHR applications. We focus on a broad scope of tasks, namely, classification and prediction, word embeddings, extraction, generation, and other topics such as question answering, phenotyping, knowledge graphs, medical dialogue, multilinguality, interpretability, etc.CCS Concepts: • General and reference → Surveys and overviews; • Computing methodologies → Natural language processing; Machine learning algorithms.

show abstract

“…Semantic textual similarity (STS), a measure of the degree of relatedness between sentence pairs, is an important text-mining research topic [ 1 ]. STS has been widely used in biomedical and clinical domains, including information retrieval (finding relevant sentences or passages [ 2 ]), biocuration (finding key sentences for evidence attribution [ 3 ]), and question answering (finding answer-snippet candidates [ 4 ]). Despite its importance, expertly annotated STS data sets are lacking in the biomedical and clinical domains.…”

Section: Introductionmentioning

confidence: 99%

Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation Study

Chen¹,

Rankine²,

Peng³

et al. 2021

JMIR Med Inform

Self Cite

View full text Add to dashboard Cite

Background Semantic textual similarity (STS) measures the degree of relatedness between sentence pairs. The Open Health Natural Language Processing (OHNLP) Consortium released an expertly annotated STS data set and called for the National Natural Language Processing Clinical Challenges. This work describes our entry, an ensemble model that leverages a range of deep learning (DL) models. Our team from the National Library of Medicine obtained a Pearson correlation of 0.8967 in an official test set during 2019 National Natural Language Processing Clinical Challenges/Open Health Natural Language Processing shared task and achieved a second rank. Objective Although our models strongly correlate with manual annotations, annotator-level correlation was only moderate (weighted Cohen κ=0.60). We are cautious of the potential use of DL models in production systems and argue that it is more critical to evaluate the models in-depth, especially those with extremely high correlations. In this study, we benchmark the effectiveness and efficiency of top-ranked DL models. We quantify their robustness and inference times to validate their usefulness in real-time applications. Methods We benchmarked five DL models, which are the top-ranked systems for STS tasks: Convolutional Neural Network, BioSentVec, BioBERT, BlueBERT, and ClinicalBERT. We evaluated a random forest model as an additional baseline. For each model, we repeated the experiment 10 times, using the official training and testing sets. We reported 95% CI of the Wilcoxon rank-sum test on the average Pearson correlation (official evaluation metric) and running time. We further evaluated Spearman correlation, R², and mean squared error as additional measures. Results Using only the official training set, all models obtained highly effective results. BioSentVec and BioBERT achieved the highest average Pearson correlations (0.8497 and 0.8481, respectively). BioSentVec also had the highest results in 3 of 4 effectiveness measures, followed by BioBERT. However, their robustness to sentence pairs of different similarity levels varies significantly. A particular observation is that BERT models made the most errors (a mean squared error of over 2.5) on highly similar sentence pairs. They cannot capture highly similar sentence pairs effectively when they have different negation terms or word orders. In addition, time efficiency is dramatically different from the effectiveness results. On average, the BERT models were approximately 20 times and 50 times slower than the Convolutional Neural Network and BioSentVec models, respectively. This results in challenges for real-time applications. Conclusions Despite the excitement of further improving Pearson correlations in this data set, our results highlight that evaluations of the effectiveness and efficiency of STS models are critical. In future, we suggest more evaluations on the generalization capability and user-level testing of the models. We call for community efforts to create more biomedical and clinical STS data sets from different perspectives to reflect the multifaceted notion of sentence-relatedness.

show abstract

Artificial Intelligence in Action: Addressing the COVID-19 Pandemic with Natural Language Processing

Cited by 50 publications

References 113 publications

A COVID-19 Search Engine (CO-SE) with Transformer-based architecture

A COVID-19 Search Engine (CO-SE) with Transformer-based architecture

Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review

Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation Study

Contact Info

Product

Resources

About