Are synthetic clinical notes useful for real natural language processing tasks: A case study on clinical entity recognition

Abstract

Objective : Developing clinical natural language processing systems often requires access to many clinical documents, which are not widely available to the public due to privacy and security concerns. To address this challenge, we propose to develop methods to generate synthetic clinical notes and evaluate their utility in real clinical natural language processing tasks. Materials and Methods : We implemented 4 state-of-the-art text generation models, namely CharRNN, SegGAN, GPT-2, and CTRL, to generate clinical text for the History and Present Illness section. We then manually annotated clinical entities for randomly selected 500 History and Present Illness notes generated from the best-performing algorithm. To compare the utility of natural and synthetic corpora, we trained named entity recognition (NER) models from all 3 corpora and evaluated their performance on 2 independent natural corpora. Results : Our evaluation shows GPT-2 achieved the best BLEU (bilingual evaluation understudy) score (with a BLEU-2 of 0.92). NER models trained on synthetic corpus generated by GPT-2 showed slightly better performance on 2 independent corpora: strict F1 scores of 0.709 and 0.748, respectively, when compared with the NER models trained on natural corpus (F1 scores of 0.706 and 0.737, respectively), indicating the good utility of synthetic corpora in clinical NER model development. In addition, we also demonstrated that an augmented method that combines both natural and synthetic corpora achieved better performance than that uses the natural corpus only. Conclusions : Recent advances in text generation have made it possible to generate synthetic clinical notes that could be useful for training NER models for information extraction from natural clinical notes, thus lowering the privacy concern and increasing data availability. Further investigation is needed to apply this technology to practice.

Privacy Protection and Secondary Use of Health Data: Strategies and Methods

“…To address this challenge, Li et al proposed to develop methods to generate synthetic clinical notes and evaluate their utility in real clinical natural language processing tasks. Thanks to the development of deep learning, recent advances in text generation have made it possible to generate synthetic clinical notes that could be useful for training NER models for information extraction from natural clinical notes, thus lowering the privacy concern and increasing data availability [ 48 ].…”

Medical Text Prediction and Suggestion Using Generative Pretrained Transformer Models with Dental Medical Notes

“…language generation. Current applications are towards creating synthetic clinical notes to increase data size 6 , generating summaries of medical trials 7 , and generating simplified clinical notes for patients 8 .…”

Leveraging the potential of synthetic text for AI in mental healthcare

“…The first one is the challenge of developing a benchmarking standard for synthetic text ( 28 , 29 ). As yet, in the clinical and mental health domains the utility of synthetic text has been evaluated for only some types of AI, such as automated diagnoses prediction ( 21 ) or named entity recognition ( 7 ). There are no best established practices or systematic criteria on how to assess synthetic data.…”

“…Resolving, or at least significantly assuaging privacy concerns will ease this data sparsity constraint. Relevant research investigates how to make this synthetic data statistically-relevant, as well as useful for algorithmic analysis and training ( 7 , 8 ). Such data show potential to improve data accessibility, model performance and eventually boost scientific progress.…”