Does Synthetic Data Generation of LLMs Help Clinical Text Mining?

Abstract

Recent advancements in large language models (LLMs) have led to the development of highly potent models like OpenAI's ChatGPT. These models have exhibited exceptional performance in a variety of tasks, such as question answering, essay composition, and code generation. However, their effectiveness in the healthcare sector remains uncertain. In this study, we seek to investigate the potential of LLMs to aid in clinical text mining by examining their ability to extract structured information from unstructured healthcare texts, with a focus on biological named entity recognition and relation extraction. However, our preliminary results indicate that employing LLMs directly for these tasks resulted in poor performance and raised privacy concerns associated with uploading patients' information to the LLM API. To overcome these limitations, we propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data with labels utilizing LLMs and fine-tuning a local model for the downstream task. Our method has resulted in significant improvements in the performance of downstream tasks, improving the F1-score from 23.37% to 63.99% for the named entity recognition task and from 75.86% to 83.59% for the relation extraction task. Furthermore, generating data using LLMs can significantly reduce the time and effort required for data collection and labeling, as well as mitigate data privacy concerns. In summary, the proposed framework presents a promising solution to enhance the applicability of LLM models to clinical text mining.

Vocabulary Matters: An Annotation Pipeline and Four Deep Learning Algorithms for Enzyme Named Entity Recognition

“…Improving LLMs for NER tasks requires at least further fine-tuning but more likely supplying these with domain-specific training data for domains that they are not trained on. In our case, not all biomedical literature is freely shareable, and it is therefore not possible to send these data to external platforms to train such models, a problem that is potentially solvable by generating synthetic data for closed systems . Another issue is available compute to train such models that, even with open LLMs such as LLaMa, require much more resources to train than BERT-based models.…”

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

“…If there is a relation, then the label should be "Yes", otherwise "No". (Tang et al, 2023) HoC document: < text>; target: The correct category for this document is ? You must choose from the given list of answer categories (introduce what each category is ...)" (Chen et al, 2023) Table 4: The prompts used for different evaluation tasks and datasets.…”

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…The research community explored GLLMs for data generation-based data augmentation in various NLP tasks like dialogue generation [410], training smaller LLMs [411], [416], common sense reasoning [412], hate speech detection [413], undesired content detection [414], question answering [415], [425], intent classification [143], relation extraction [155], [422], instruction tuning [417], [418], paraphrase detection [420], tweet intimacy prediction [421], named entity recognition [422], machine translation [424] etc. GLLM-based data generation for data augmentation is explored in multiple domains like general [143], [155], [412], [416]- [418], [420], [424]- [426], social media [409], [413], [414], [421], [423], news [423], scientific literature [155], [420], healthcare [410], [415], [422], dialogue [419], programming [411] etc. Table 19 presents a summary of research works exploring GLLMs for data generationbased data augmentation.…”

“…Based on the evaluation on four topic classification datasets, the authors observed that (i) the proposed approach enhances the model performance and (ii) reduces the querying cost of ChatGPT by a large margin. Some of the research works explored GLLMs for data generation-based data augmentation in various information extraction tasks like relation extraction [155], relation classification [422] and named entity recognition [422]. Xu et al [155] evaluated how effective is the GPT-3.5 model for relation classification.…”

“…The prompt used for data generation consists of instance descriptions along with some example instances. Tang et al [422] used ChatGPT in zero-shot settings to generate synthetic data for tasks like named entity recognition and relation classification in the healthcare domain. The authors showed that the model fine-tuned on this synthetic data outperforms zero-shot ChatGPT by a large margin in both tasks.…”