Saama Research at MEDIQA 2019: Pre-trained BioBERT with Attention Visualisation for Medical Natural Language Inference

Abstract

Natural Language inference is the task of identifying relation between two sentences as entailment, contradiction or neutrality. MedNLI is a biomedical flavour of NLI for clinical domain. This paper explores the use of Bidirectional Encoder Representation from Transformer (BERT) for solving MedNLI. The proposed model, BERT pre-trained on PMC, PubMed and fine-tuned on MIMIC-III v1.4, achieves state of the art results on MedNLI (83.45%) and an accuracy of 78.5% in MEDIQA challenge. The authors present an analysis of the attention patterns that emerged as a result of training BERT on MedNLI using a visualization tool, bertviz. * *Equal Contribution: Kamal had sole access to MIMIC and MEDIQA data, focussed on the algorithm development and implementation. Suriyadeepan and Archana focussed on the attention visualisation and writing. Soham and Malaikannan focussed on reviewing

Enriching Biomedical Knowledge for Low-resource Language Through Translation

“…The source of premise sentences in MedNLI is from MIMIC-III (Johnson et al, 2016), a large open-source clinical database. The dataset has been widely studied and benchmarked by the Biomedical NLP research community 11 (Peng et al, 2019; Phan et al, 2021a; El Boukkouri et al, 2020; Alrowili and Shanker, 2021; Kanakarajan et al, 2019).…”

Low Probability of Detection Communication: Opportunities and Challenges

“…To incorporate unstructured knowledge into the stance detection models, WS-BERT [4] directly infuses external knowledge from Wikipedia as its inputs to pre-trained models for stance detection in the VAST dataset. Another knowledge infusion paradigm is that finetune PLMs on the specific domain corpus to embed the domain-specific knowledge, as demonstrated by Sci-BERT [17], Bio-BERT [18], BERTweet [19]. In addition to domain-specific finetuning, Self-talk [20] offers another interesting solution by exploring knowledge from its own training corpus with hand-crafted prompts, enhancing language model learning with task-related knowledge.…”