Are we there yet? Exploring clinical domain knowledge of BERT models

Abstract

We explore whether state-of-the-art BERT models encode sufficient domain knowledge to correctly perform domain-specific inference. Although BERT implementations such as BioBERT are better at domain-based reasoning than those trained on general-domain corpora, there is still a wide margin compared to human performance on these tasks. To bridge this gap, we explore whether supplementing textual domain knowledge in the medical NLI task: a) by further language model pretraining on the medical domain corpora, b) by means of lexical match algorithms such as the BM25 algorithm, c) by supplementing lexical retrieval with dependency relations, or d) by using a trained retriever module, can push this performance closer to that of humans. We do not find any significant difference between knowledge supplemented classification as opposed to the baseline BERT models, however. This is contrary to the results for evidence retrieval on other tasks such as open domain question answering (QA). By examining the retrieval output, we show that the methods fail due to unreliable knowledge retrieval for complex domain-specific reasoning. We conclude that the task of unsupervised text retrieval to bridge the gap in existing information to facilitate inference is more complex than what the state-of-the-art methods can solve, and warrants extensive research in the future.

Phenotyping in clinical text with unsupervised numerical reasoning for patient stratification

“…Overall, although these models show benefits in their respective domains, they did not incorporate clinical knowledge to address challenges in clinical applications. 23 In practice, NR for clinical context has the following challenges. First, there can be accumulation of multiple numeric examples in a condensed context, such as "Physical examination: temperature 97.5, blood pressure 124/55, pulse 79, respirations 18, O 2 saturation 99% on room air."…”

Interpreting Patient Descriptions using Distantly Supervised Similar Case Retrieval

“…More generally, however, there is some evidence that the effectiveness of augmenting questions with textual knowledge is limited in the biomedical domain. For instance, Sushil et al [62] evaluated the effect of such augmentation strategies and failed to obtain any statistically significant improvements for MedNLI [55], a well-known benchmark for Natural Language Inference (NLI) in the biomedical domain. These findings were also corroborated by our own initial analysis.…”

“…When it comes to interpreting patient descriptions, however, the potential of such strategies is less clear. For instance, Sushil et al [62] used an information retrieval engine to find relevant sentences in biomedical corpora, which were then added to the premise of Natural Language Inference (NLI) instances. In experiments on MedNLI [55], they found no statistically significant improvements as a result Table 1: Example of a question from MedQA, along with the answer candidates.…”

Unsupervised Numerical Reasoning to Extract Phenotypes from Clinical Text by Leveraging External Knowledge

“…Other works [8,12,28] designed special modules for numerical reasoning in text which were then integrated with neural networks. Overall, these models have shown advancements in the respective domains for specialized problems but they did not incorporate clinical knowledge with specific extensive reasoning for clinical applications [33].…”