How to Pre-train Your Model? Comparison of Different Pre-training Models for Biomedical Question Answering

Kamath, Sanjay; Grau, Brigitte; Ma, Yue

doi:10.1007/978-3-030-43887-6_58

Cited by 7 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, WordPiece vocabulary needs to be constructed based on biomedical corpora. The reasons for the use of the original BERT base vocabulary are as follows: first to make BioBERT compatible with the BERT model, which helps BERT to be reused pretrained on general domain corpora, making current BERT simpler to use; and second is that any new terms for the biomedical domain can be interpreted and fine-tuned using the original BERT WordPiece vocabulary (Kamath et al , 2020).…”

Section: Methodsmentioning

confidence: 99%

ABEE: automated bio entity extraction from biomedical text documents

Kumar

Sharaff

2023

DTA

View full text Add to dashboard Cite

PurposeThe purpose of this study was to design a multitask learning model so that biomedical entities can be extracted without having any ambiguity from biomedical texts.Design/methodology/approachIn the proposed automated bio entity extraction (ABEE) model, a multitask learning model has been introduced with the combination of single-task learning models. Our model used Bidirectional Encoder Representations from Transformers to train the single-task learning model. Then combined model's outputs so that we can find the verity of entities from biomedical text.FindingsThe proposed ABEE model targeted unique gene/protein, chemical and disease entities from the biomedical text. The finding is more important in terms of biomedical research like drug finding and clinical trials. This research aids not only to reduce the effort of the researcher but also to reduce the cost of new drug discoveries and new treatments.Research limitations/implicationsAs such, there are no limitations with the model, but the research team plans to test the model with gigabyte of data and establish a knowledge graph so that researchers can easily estimate the entities of similar groups.Practical implicationsAs far as the practical implication concerned, the ABEE model will be helpful in various natural language processing task as in information extraction (IE), it plays an important role in the biomedical named entity recognition and biomedical relation extraction and also in the information retrieval task like literature-based knowledge discovery.Social implicationsDuring the COVID-19 pandemic, the demands for this type of our work increased because of the increase in the clinical trials at that time. If this type of research has been introduced previously, then it would have reduced the time and effort for new drug discoveries in this area.Originality/valueIn this work we proposed a novel multitask learning model that is capable to extract biomedical entities from the biomedical text without any ambiguity. The proposed model achieved state-of-the-art performance in terms of precision, recall and F1 score.

show abstract

Section: Methodsmentioning

confidence: 99%

ABEE: automated bio entity extraction from biomedical text documents

Kumar

Sharaff

2023

DTA

View full text Add to dashboard Cite

show abstract

“…PubMedBERT [66], BioMegatron [215], Yoon et al [284], Jeong et al [89], Chakraborty et al [30], Kamath et al [100], Du et al [52], Yoon et al [283], Zhou et al [300], Akdemir et al [5], He et al [78], Amherst et al [200], Kommaraju et al [112], for COVID-19 [55,120,170,201], Soni et al [222], Mairittha et al [147]. Dialogue Systems Zeng at al.…”

Section: Question Answeringmentioning

confidence: 99%

“…BioMedBERT is based on the BERT model pre-trained on a large-scale biomedical literature dataset BREATHE. Kamath et al [100] compared the effectiveness of pre-trained models for machine-reading comprehension and question-answering in the general domain in fine-tuning the biomedical question answering task. They found that the question answering model fits better to the task.…”

Section: Question Answeringmentioning

confidence: 99%

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Wang¹,

Xie²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities proposes various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.CCS Concepts: • Computing methodologies → Natural language processing; Natural language generation; Neural networks; Bio-inspired approaches.

show abstract

“…As the language modelling can be seen as independent tasks, some researchers see pre-training with language modelling objective as a part of the transfer learning paradigm [9,6]. Pre-training on natural language understanding tasks, in particular, on sentence modelling tasks, help not only to improve the quality of the task under consideration [2,21,12], but also to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity [19].…”

Section: Related Workmentioning

confidence: 99%

DaNetQA: A Yes/No Question Answering Dataset for the Russian Language

Глушкова

Machnev

Fenogenova

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

DaNetQA, a new question-answering corpus, follows BoolQ [2] design: it comprises natural yes/no questions. Each question is paired with a paragraph from Wikipedia and an answer, derived from the paragraph. The task is to take both the question and a paragraph as input and come up with a yes/no answer, i.e. to produce a binary output. In this paper, we present a reproducible approach to DaNetQA creation and investigate transfer learning methods for task and language transferring. For task transferring we leverage three similar sentence modelling tasks: 1) a corpus of paraphrases, Paraphraser, 2) an NLI task, for which we use the Russian part of XNLI, 3) another question answering task, SberQUAD. For language transferring we use English to Russian translation together with multilingual language fine-tuning.

show abstract

How to Pre-train Your Model? Comparison of Different Pre-training Models for Biomedical Question Answering

Cited by 7 publications

References 17 publications

ABEE: automated bio entity extraction from biomedical text documents

ABEE: automated bio entity extraction from biomedical text documents

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

DaNetQA: A Yes/No Question Answering Dataset for the Russian Language

Contact Info

Product

Resources

About