Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang, Benyou; Xie, Qianqian; Pei, Jiahuan; Tiwari, Prayag; Zhao, Li; Fu, Jie

doi:10.48550/arxiv.2110.05006

Cited by 12 publications

(13 citation statements)

References 288 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…domain such as BioBERT , Clini-calXLNET (Huang et al, 2020), and others (Alrowili and Shanker, 2021; Kraljevic et al, 2021;Phan et al, 2021), this paradigm is widely used for creating many task-specific models (Wang et al, 2021a;Banerjee et al, 2021). However, task-specific models have limitations to real-world applications because this approach is computationally expensive (i.e., require large computational resources) and time-consuming (Strubell et al, 2019;Schwartz et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Parmar¹,

Mishra²,

Purohit³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

Single-task models have proven pivotal in solving specific tasks; however, they have limitations in real-world applications where multitasking is necessary and domain shifts are exhibited. Recently, instructional prompts have shown significant improvement towards multitask generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain. Motivated by this, this paper explores the impact of instructional prompts for biomedical MTL. We introduce the BoX, a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Using this meta-dataset, we propose a unified model termed as In-BoXBART, that can jointly learn all tasks of the BoX without any task-specific modules. To the best of our knowledge, this is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks. Experimental results indicate that the proposed model: 1) outperforms single-task baseline by ∼3% and multitask (without instruction) baseline by ∼18% on an average, and 2) shows ∼23% improvement compared to single-task baseline in few-shot learning (i.e., 32 instances per task) on an average. Our analysis indicates that there is significant room for improvement across tasks in the BoX, implying the scope for future research direction. 1

show abstract

Section: Introductionmentioning

confidence: 99%

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Parmar¹,

Mishra²,

Purohit³

et al. 2022

Findings of the Association for Computational Linguistics: NAACL 2022

View full text Add to dashboard Cite

show abstract

“…3) Our dataset only includes papers in the biomedical domain, while SSN consists of papers from several different fields including mathematical, physics, and computer science, our dataset can help the evaluation of domainspecific tasks in the research community. Moreover, biomedical scientific papers are laden with terminology and have complex syntactic structures [21,31]. This makes our dataset a challenging benchmark for automatic summarization methods.…”

Section: Citation-aware Pubmed Datasetmentioning

confidence: 99%

CitationSum: Citation-aware Graph Contrastive Learning for Scientific Paper Summarization

Luo¹,

Ananiadou²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…[42], [43] as an image extraction approach, following the recent breakthrough of Transformers [27] in handling natural language processing tasks [44]. ViT mainly consists of the following parts: Linear Projection of Flattened Patches (Embedding layer), Transformer Encoder, and MLP Head.…”

Section: Sota Vit and Cnn Modelsmentioning

confidence: 99%

STCovidNet: Automatic Detection Model of Novel Coronavirus Pneumonia Based on Swin Transformer

Wang

Zhang

Tian

2022

Preprint

View full text Add to dashboard Cite

The novel coronavirus disease 2019 (COVID-19) has emerged as an enormous challenge facing China today. Preventive Medicine physicians and Artificial Intelligence (AI) researchers try to improve the ability to early automatic warning of coronavirus infections, promote epidemic prevention, and reduce medical costs using deep learning methods. In this work, we build an extensive database of chest computed tomography (CT) scans with image data from domestic and international open-source medical datasets. Swin Transformer is chosen as the backbone network to establish a model (STCovidNet) for the prediction of COVID-19. We then compare the performance of our technique against that of Vision Transformer (ViT) and Convolutional Neural Network (CNN). Next, to visualize our model's high-dimensional outputs in 2-dimensional space, we apply t-distributed stochastic neighbor embedding (t-SNE) as the dimension-reduction strategy. Finally, we employ gradient-weighted class activation mapping (Grad-CAM) to present a class activation map. The results indicate that STCovidNet’s performance surpasses ViT and CNN with a 0.9811 AUC and 0.9858 accuracy score. Our network outperforms previous techniques to reduce intra-class variability and generate well-separated feature embedding. The CAM figure illustrates that the decision region corresponds to radiologists' detecting spots. The suggested method can be an effective way of catching COVID-19 instances.

show abstract

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Cited by 12 publications

References 288 publications

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

CitationSum: Citation-aware Graph Contrastive Learning for Scientific Paper Summarization

STCovidNet: Automatic Detection Model of Novel Coronavirus Pneumonia Based on Swin Transformer

Contact Info

Product

Resources

About