Fine-tuning large neural language models for biomedical natural language processing

Tinn, Robert; Cheng, Hao; Gu, Yu; Usuyama, Naoto; Liu, Xiaodong; Naumann, Tristan; Gao, Jianfeng; Poon, Hoifung

doi:10.1016/j.patter.2023.100729

Cited by 42 publications

(17 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The overall low mDISCERN scores observed for the Bloomz model should not be interpreted as a definitive disqualification for patient recommendation. Instead, these findings should motivate the scientific community to explore and enhance the potential of this model through advanced fine-tuning techniques [25] and more effective prompting strategies, especially given that it is the sole open-source model within the examined cohort. Other general factors that might have an impact on model performance are the complexity and diversity of the training data, the presence of inherent biases in the data, the computational resources available during training, general model architecture, and the ongoing adjustments and updates made to the model postdeployment to respond to real-world feedback.…”

Section: Discussionmentioning

confidence: 99%

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

Wilhelm,

Roos,

Kaczmarczyk

2023

J Med Internet Res

View full text Add to dashboard Cite

Background As advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties. Objective This study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology. Methods Three domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI’s most advanced model, GPT-4, an automated evaluation of each model’s responses to the diseases was performed using the same criteria and compared to the physicians’ assessments through Pearson correlation analysis. Results Claude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P<.001). Evaluating their reliability, the models displayed strong contrasts in their falseness ratings, with variations both across models (P<.001) and specialties (P<.001). Distinct error patterns emerged, such as confusing diagnoses; providing vague, ambiguous advice; or omitting critical treatments, such as antibiotics for infectious diseases. Regarding potential harm, GPT-3.5-Turbo was found to be the safest, with the lowest harmfulness rating. All models lagged in detailing the risks associated with treatment procedures, explaining the effects of therapies on quality of life, and offering additional sources of information. Pearson correlation analysis underscored a substantial alignment between physician assessments and GPT-4’s evaluations across all established criteria (P<.01). Conclusions This study, while comprehensive, was limited by the involvement of a select number of specialties and physician evaluators. The straightforward prompting strategy (“How to treat…”) and the assessment benchmarks, initially conceptualized for human-authored content, might have potential gaps in capturing the nuances of AI-driven information. The LLMs evaluated showed a notable capability in generating valuable medical content; however, evident lapses in content quality and potential harm signal the need for further refinements. Given the dynamic landscape of LLMs, this study’s findings emphasize the need for regular and methodical assessments, oversight, and fine-tuning of these AI tools to ensure they produce consistently trustworthy and clinically safe medical advice. Notably, the introduction of an auto-evaluation mechanism using GPT-4, as detailed in this study, provides a scalable, transferable method for domain-agnostic evaluations, extending beyond therapy recommendation assessments.

show abstract

Section: Discussionmentioning

confidence: 99%

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

Wilhelm,

Roos,

Kaczmarczyk

2023

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…This operation requires a costly manual curation, operated by highly expert users. This is an inevitable effort to handle data scarcity, analyzed in [ 65 ] in general terms, becoming even more relevant in biomedical fields [ 66 , 67 ]. To minimize such effort in our case, we implemented a process that supports the building of small high-quality training datasets.…”

Section: Methodsmentioning

confidence: 99%

CoVEffect: interactive system for mining the effects of SARS-CoV-2 mutations and variants based on deep learning

Serna García,

Al Khalaf,

Invernici

et al. 2022

GigaScience

View full text Add to dashboard Cite

Background Literature about SARS-CoV-2 widely discusses the effects of variations that have spread in the past 3 years. Such information is dispersed in the texts of several research articles, hindering the possibility of practically integrating it with related datasets (e.g., millions of SARS-CoV-2 sequences available to the community). We aim to fill this gap, by mining literature abstracts to extract—for each variant/mutation—its related effects (in epidemiological, immunological, clinical, or viral kinetics terms) with labeled higher/lower levels in relation to the nonmutated virus. Results The proposed framework comprises (i) the provisioning of abstracts from a COVID-19–related big data corpus (CORD-19) and (ii) the identification of mutation/variant effects in abstracts using a GPT2-based prediction model. The above techniques enable the prediction of mutations/variants with their effects and levels in 2 distinct scenarios: (i) the batch annotation of the most relevant CORD-19 abstracts and (ii) the on-demand annotation of any user-selected CORD-19 abstract through the CoVEffect web application (http://gmql.eu/coveffect), which assists expert users with semiautomated data labeling. On the interface, users can inspect the predictions and correct them; user inputs can then extend the training dataset used by the prediction model. Our prototype model was trained through a carefully designed process, using a minimal and highly diversified pool of samples. Conclusions The CoVEffect interface serves for the assisted annotation of abstracts, allowing the download of curated datasets for further use in data integration or analysis pipelines. The overall framework can be adapted to resolve similar unstructured-to-structured text translation tasks, which are typical of biomedical domains.

show abstract

“…Recent studies have provided compelling evidence that the fine-tuning of Large Language Models (LLMs) with specialized medical data sources can facilitate their adaptation to specific tasks within clinical settings [19, 20, 21]. For instance, Yang et al successfully developed an LLM model from fine-tuning BERT and ChatGPT to extract and recognize HPO phenotypes in clinical texts within the presence of non-HPO phenotypes, typos, and semantic differences with the model’s original training data [22].…”

Section: Background and Significancementioning

confidence: 99%

“…Recent studies have provided compelling evidence that the fine-tuning of LLMs with specialized medical data sources can facilitate their adaptation to specific tasks within clinical settings [20][21][22]. Fine-tuning is a process in which an unsupervised pre-trained LLM is further trained on a smaller, task-specific dataset to adapt its parameters to a specific task.…”

Section: Introductionmentioning

confidence: 99%

Fine-tuning Large Language Models for Rare Disease Concept Normalization

Wang,

Liu,

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

Objective: We aim to develop a solution for rare disease concept normalization based on fine-tuning LLaMA 2, an open-source large language model (LLM), using a domain-specific corpus. Methods and Materials: We fine-tuned four LLaMA2 models, each comprising seven billion parameters, using sentences incorporating clinical concepts from the HPO and OMIM vocabularies. The fine-tuning was conducted on four NVIDIA A100 GPUs. Results: All models proved resilient to newly prompt-engineered sentences not used in the fine-tuning, achieved nearly perfect accuracies when prompted with original training data, and exhibit some robustness to typos. We tested each model on concepts they had not been trained on. The non-synonym HPO model fine-tuned without synonyms achieved 25.2% accuracy, while the synonym HPO model, fine-tuned with half the synonyms, achieved 85.6% accuracy. When tested against concept synonyms from SNOMED-CT, the non-synonym model achieved an accuracy of 33.9% while the synonym model improved to 57.4%. Synonyms proved challenging to both non-synonym and synonym OMIM models. ChatGPT 3.5 correctly identified HPO IDs for four out of 20 prompts. Discussion: Our increasingly fine-tuned models demonstrated growing robustness to challenges such as misspellings, synonyms, and concepts from other ontologies. Incorrect outputs stem from tokens in the input that the models have never encountered, such as parenthesis. Many synonyms do not share the same semantic meaning and often include abbreviations. Conclusion: Our fine-tuned LLaMA 2 models provide the capability to identify variations in medical concepts from clinical narratives while successfully normalizing them to a standard concept.

show abstract

Fine-tuning large neural language models for biomedical natural language processing

Cited by 42 publications

References 14 publications

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study

CoVEffect: interactive system for mining the effects of SARS-CoV-2 mutations and variants based on deep learning

Fine-tuning Large Language Models for Rare Disease Concept Normalization

Contact Info

Product

Resources

About