Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

Gutiérrez, Bernal Jiménez; Nikolas, McNeal,; Washington, Clayton; Chen, You; Li, Lang; Sun, Hongjian; Sui, Yu

doi:10.18653/v1/2022.findings-emnlp.329

Cited by 31 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This signals the need to improve the span extraction which currently limits the overall performance of our system with a lost of 10.3 points (0.82 -0.717) as shown in Table 2 when comparing the performances achieved by PhenoID -Noisy-BIO and by PhenoID (with gold truth spans given). We are exploring the benefit of large language models (LLMs) to replace our sequence labeler with a generative model, whereby we present an observation in a prompt and ask the model to list all HPO terms mentioned in the observation as suggested in previous studies on different tasks [18].…”

Section: Resultsmentioning

confidence: 99%

PhenoID, a language model normalizer of physical examinations from genetics clinical notes

Weissenbacher,

Rawal,

Zhao

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Phenotypes identified during dysmorphology physical examinations are critical to genetic diagnosis and nearly universally documented as free-text in the electronic health record (EHR). Variation in how phenotypes are recorded in free-text makes large-scale computational analysis extremely challenging. Existing natural language processing (NLP) approaches to address phenotype extraction are trained largely on the biomedical literature or on case vignettes rather than actual EHR data. Methods: We implemented a tailored system at the Children's Hospital of Philadelpia that allows clinicians to document dysmorphology physical exam findings. From the underlying data, we manually annotated a corpus of 3136 organ system observations using the Human Phenotype Ontology (HPO). We provide this corpus publicly. We trained a transformer based NLP system to identify HPO terms from exam observations. The pipeline includes an extractor, which identifies tokens in the sentence expected to contain an HPO term, and a normalizer, which uses those tokens together with the original observation to determine the specific term mentioned. Findings: We find that our labeler and normalizer NLP pipeline, which we call PheNorm, achieves state-of-the-art performance for the dysmorphology physical exam phenotype extraction task. PheNorm's performance on the test set was 0.717, compared to the nearest baseline system (PhenoTagger) performance of 0.633. An analysis of our system's normalization errors shows possible imperfections in the HPO terminology itself but also reveals a lack of semantic understanding by our transformer models. Interpretation: Transformers-based NLP models are a promising approach to genetic phenotype extraction and, with recent development of larger pre-trained causal language models, may improve semantic understanding in the future. We believe our results also have direct applicability to more general extraction of medical signs and symptoms.

show abstract

Section: Resultsmentioning

confidence: 99%

PhenoID, a language model normalizer of physical examinations from genetics clinical notes

Weissenbacher,

Rawal,

Zhao

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A solution could be to retrieve the paragraphs documenting a GOC discussion with a generative model like chat-GPT Ouyang et al (2022). However, we left the evaluation of this approach as future work since large language models need to be fined-tuned on clinical notes to achieve competing performance Jimenez Gutierrez et al (2022), a process that is still an economic and technical challenge Scao et al (2022).…”

Section: Discussionmentioning

confidence: 99%

Detecting Goals of Care Conversations in Clinical Notes with Active Learning

Weissenbacher,

Courtright,

Rawal

et al. 2024

Preprint

View full text Add to dashboard Cite

ObjectiveGoals Of Care (GOC) discussions are an increasingly used quality metric in serious illness care and research. Wide variation in documentation practices within the Electronic Health Record (EHR) presents challenges for reliable measurement of GOC discussions. Novel natural language processing approaches are needed to capture GOC discussions documented in real-world samples of seriously ill hospitalized patients’ EHR notes, a corpus with a very low event prevalence.MethodsTo automatically detect utterances documenting GOC discussions outside of dedicated GOC note types, we proposed an ensemble of classifiers aggregating the predictions of rule-based, feature-based, and three transformers-based classifiers. We trained our classifier on 600 manually annotated EHR notes among patients with serious illnesses. Our corpus exhibited an extremely imbalanced ratio between utterances discussing GOC and utterances that do not. This ratio challenges standard supervision methods to train a classifier. Therefore, we trained our classifier with active learning.ResultsUsing active learning, we reduced the annotation cost to fine-tune our ensemble by 70% while improving its performance in our test set of 176 EHR notes, with 0.557 F1-score for utterance classification and 0.629 for note classification.ConclusionWhen classifying notes, with a true positive rate of 72% (13/18) and false positive rate of 8% (13/158), our performance may be sufficient for deploying our classifier in the EHR to facilitate point-of-care access to GOC conversations documented outside of dedicated notes types, without overburdening clinicians with false positives. Improvements are needed before using it to enrich trial populations or as an outcome measure.

show abstract

“…The objective was to ensure that the extracted candidate PHI text could be matched with the original input text represented by the "<SENTENCE>" placeholder. Following the prompt design procedure [49], we organized our prompt into 4 main parts:…”

Section: Chatgpt-based Deidentification Frameworkmentioning

confidence: 99%

Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study

Lee,

Chen,

Chen

et al. 2024

J Med Internet Res

View full text Add to dashboard Cite

Background The widespread use of electronic health records in the clinical and biomedical fields makes the removal of protected health information (PHI) essential to maintain privacy. However, a significant portion of information is recorded in unstructured textual forms, posing a challenge for deidentification. In multilingual countries, medical records could be written in a mixture of more than one language, referred to as code mixing. Most current clinical natural language processing techniques are designed for monolingual text, and there is a need to address the deidentification of code-mixed text. Objective The aim of this study was to investigate the effectiveness and underlying mechanism of fine-tuned pretrained language models (PLMs) in identifying PHI in the code-mixed context. Additionally, we aimed to evaluate the potential of prompting large language models (LLMs) for recognizing PHI in a zero-shot manner. Methods We compiled the first clinical code-mixed deidentification data set consisting of text written in Chinese and English. We explored the effectiveness of fine-tuned PLMs for recognizing PHI in code-mixed content, with a focus on whether PLMs exploit naming regularity and mention coverage to achieve superior performance, by probing the developed models’ outputs to examine their decision-making process. Furthermore, we investigated the potential of prompt-based in-context learning of LLMs for recognizing PHI in code-mixed text. Results The developed methods were evaluated on a code-mixed deidentification corpus of 1700 discharge summaries. We observed that different PHI types had preferences in their occurrences within the different types of language-mixed sentences, and PLMs could effectively recognize PHI by exploiting the learned name regularity. However, the models may exhibit suboptimal results when regularity is weak or mentions contain unknown words that the representations cannot generate well. We also found that the availability of code-mixed training instances is essential for the model’s performance. Furthermore, the LLM-based deidentification method was a feasible and appealing approach that can be controlled and enhanced through natural language prompts. Conclusions The study contributes to understanding the underlying mechanism of PLMs in addressing the deidentification process in the code-mixed context and highlights the significance of incorporating code-mixed training instances into the model training phase. To support the advancement of research, we created a manipulated subset of the resynthesized data set available for research purposes. Based on the compiled data set, we found that the LLM-based deidentification method is a feasible approach, but carefully crafted prompts are essential to avoid unwanted output. However, the use of such methods in the hospital setting requires careful consideration of data security and privacy concerns. Further research could explore the augmentation of PLMs and LLMs with external knowledge to improve their strength in recognizing rare PHI.

show abstract

Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

Cited by 31 publications

References 0 publications

PhenoID, a language model normalizer of physical examinations from genetics clinical notes

PhenoID, a language model normalizer of physical examinations from genetics clinical notes

Detecting Goals of Care Conversations in Clinical Notes with Active Learning

Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study

Contact Info

Product

Resources

About