AMMU: A survey of transformer-based biomedical pretrained language models

Kalyan, Katikapalli Subramanyam; Rajasekharan, Ajit; Sangeetha, S.

doi:10.1016/j.jbi.2021.103982

Cited by 125 publications

(66 citation statements)

References 143 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning and artificial intelligence have been increasingly applied in various domain such as computer vision 51,52 , natural language processing [53][54][55] , drug discovery 56,57 , QSAR [58][59][60] , and genomics [61][62][63] . AI methods such as convolutional neural networks (CNNs) 64 and recurrent neural networks (RNNs) 65 that are extensively used in computer vision and natural language processing have been investigated for identifying protein binding sites in DNA and RNA sequences, and achieved state-of-the-art performance [66][67][68] .…”

Section: Discussionmentioning

confidence: 99%

Artificial immune cell,AI-cell, a new tool to predict interferon production by peripheral blood monocytes in response to nucleic acid nanoparticles

Chandler

Jain

Halman

et al. 2022

Preprint

View full text Add to dashboard Cite

Nucleic acid nanoparticles, or NANPs, are rationally designed to communicate with the human immune system and can offer innovative therapeutic strategies to overcome the limitations of traditional nucleic acid therapies. Each set of NANPs is unique in their architectural parameters and physicochemical properties, which together with the type of delivery vehicles determine the kind and the magnitude of their immune response. Currently, there are no predictive tools that would reliably guide NANPs design to the desired immunological outcome, a step crucial for the success of personalized therapies. Through a systematic approach investigating physicochemical and immunological profiles of a comprehensive panel of various NANPs, our research team has developed a computational model based on the transformer architecture able to predict the immune activities of NANPs via construction of so-called artificial immune cell, or AI-cell. The AI-cell will aid addressing in timely manner the current critical public health challenges related to overdose and safety criteria of nucleic acid therapies and promote the development of novel biomedical tools.

show abstract

Section: Discussionmentioning

confidence: 99%

Artificial immune cell,AI-cell, a new tool to predict interferon production by peripheral blood monocytes in response to nucleic acid nanoparticles

Chandler

Jain

Halman

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another emerging area is exploring generalized zero-shot learning (GZSL) [ 36 ] where the training classes are presented only at test time. Further, the performance of domain-specific LMs can be improved by reducing biases and injecting human-curated knowledge bases [ 37 ].…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT

et al. 2022

View full text Add to dashboard Cite

Background The abundance of biomedical text data coupled with advances in natural language processing (NLP) is resulting in novel biomedical NLP (BioNLP) applications. These NLP applications, or tasks, are reliant on the availability of domain-specific language models (LMs) that are trained on a massive amount of data. Most of the existing domain-specific LMs adopted bidirectional encoder representations from transformers (BERT) architecture which has limitations, and their generalizability is unproven as there is an absence of baseline results among common BioNLP tasks. Results We present 8 variants of BioALBERT, a domain-specific adaptation of a lite bidirectional encoder representations from transformers (ALBERT), trained on biomedical (PubMed and PubMed Central) and clinical (MIMIC-III) corpora and fine-tuned for 6 different tasks across 20 benchmark datasets. Experiments show that a large variant of BioALBERT trained on PubMed outperforms the state-of-the-art on named-entity recognition (+ 11.09% BLURB score improvement), relation extraction (+ 0.80% BLURB score), sentence similarity (+ 1.05% BLURB score), document classification (+ 0.62% F1-score), and question answering (+ 2.83% BLURB score). It represents a new state-of-the-art in 5 out of 6 benchmark BioNLP tasks. Conclusions The large variant of BioALBERT trained on PubMed achieved a higher BLURB score than previous state-of-the-art models on 5 of the 6 benchmark BioNLP tasks. Depending on the task, 5 different variants of BioALBERT outperformed previous state-of-the-art models on 17 of the 20 benchmark datasets, showing that our model is robust and generalizable in the common BioNLP tasks. We have made BioALBERT freely available which will help the BioNLP community avoid computational cost of training and establish a new set of baselines for future efforts across a broad range of BioNLP tasks.

show abstract

“…Previously proposed systems ( 7 , 25 ) provide scarce if not no interpretability at all. Instead, with GeMI we have made a first step toward achieving interpretability, which is paired with a functioning and effective system.…”

Section: Related Workmentioning

confidence: 99%

“…Few studies have addressed the problem of making the results explainable ( 35 ). In comparison, less approaches have employed transformed-based techniques for biomedical text extraction tasks ( 25 ), mainly focusing on entity relations ( 36 , 37 ). To the best of our knowledge, transformed-based approaches applied to biomedical tasks have not yet been combined with explainability approaches, as proposed in this article.…”

Section: Related Workmentioning

confidence: 99%

GeMI: interactive interface for transformer-based Genomic Metadata Integration

et al. 2022

View full text Add to dashboard Cite

The Gene Expression Omnibus (GEO) is a public archive containing >4 million digital samples from functional genomics experiments collected over almost two decades. The accompanying metadata describing the experiments suffer from redundancy, inconsistency and incompleteness due to the prevalence of free text and the lack of well-defined data formats and their validation. To remedy this situation, we created Genomic Metadata Integration (GeMI; http://gmql.eu/gemi/), a web application that learns to automatically extract structured metadata (in the form of key-value pairs) from the plain text descriptions of GEO experiments. The extracted information can then be indexed for structured search and used for various downstream data mining activities. GeMI works in continuous interaction with its users. The natural language processing transformer-based model at the core of our system is a fine-tuned version of the Generative Pre-trained Transformer 2 (GPT2) model that is able to learn continuously from the feedback of the users thanks to an active learning framework designed for the purpose. As a part of such a framework, a machine learning interpretation mechanism (that exploits saliency maps) allows the users to understand easily and quickly whether the predictions of the model are correct and improves the overall usability. GeMI’s ability to extract attributes not explicitly mentioned (such as sex, tissue type, cell type, ethnicity and disease) allows researchers to perform specific queries and classification of experiments, which was previously possible only after spending time and resources with tedious manual annotation. The usefulness of GeMI is demonstrated on practical research use cases. Database URL http://gmql.eu/gemi/

show abstract

AMMU: A survey of transformer-based biomedical pretrained language models

Cited by 125 publications

References 143 publications

Artificial immune cell,AI-cell, a new tool to predict interferon production by peripheral blood monocytes in response to nucleic acid nanoparticles

Artificial immune cell,AI-cell, a new tool to predict interferon production by peripheral blood monocytes in response to nucleic acid nanoparticles

Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT

GeMI: interactive interface for transformer-based Genomic Metadata Integration

Contact Info

Product

Resources

About