Probing Pre-Trained Language Models for Disease Knowledge

Abstract

Pre-trained language models such as Clini-calBERT have achieved impressive results on tasks such as medical Natural Language Inference. At first glance, this may suggest that these models are able to perform medical reasoning tasks, such as mapping symptoms to diseases. However, we find that standard benchmarks such as MedNLI contain relatively few examples that require such forms of reasoning. To better understand the medical reasoning capabilities of existing language models, in this paper we introduce DisKnE, a new benchmark for Disease Knowledge Evaluation. To construct this benchmark, we annotated each positive MedNLI example with the types of medical reasoning that are needed. We then created negative examples by corrupting these positive examples in an adversarial way. Furthermore, we define training-test splits per disease, ensuring that no knowledge about test diseases can be learned from the training data, and we canonicalize the formulation of the hypotheses to avoid the presence of artefacts. This leads to a number of binary classification problems, one for each type of reasoning and each disease. When analysing pre-trained models for the clinical/biomedical domain on the proposed benchmark, we find that their performance drops considerably.

Construction Artifacts in Metaphor Identification Datasets

“…In NLI, given a hypothesis and a premise, a system must determine whether the premise entails, contradicts or is neutral with respect to the hypothesis. Previous works have shown that supervised models could rely on superficial factors, e.g., in the SNLI dataset (Bowman et al, 2015), hypothesis-only models are surprisingly competitive (Poliak et al, 2018;Gururangan et al, 2018), a trend also observed in medical NLI datasets (Alghanmi et al, 2021).…”

Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models

“…Probing factual knowledge in PLMs Since first proposed by LAMA (Petroni et al, 2019), promptbased probing has become the main technique to assess factual knowledge in PLMs (Davison et al, 2019;Bouraoui et al, 2020;Shin et al, 2020;Brown et al, 2020;Alghanmi et al, 2021;. Given the knowledge represented in a tuple (subject, relation, object), a query q is formed by filling the subject into a relation-specific template, which is fed into the PLM.…”

“…Large-scale Pre-trained Language Models (PLMs) have demonstrated powerful capabilities in tasks where factual knowledge plays an important role (Roberts et al, 2020;. While most previous work on probing factual knowledge in PLMs has focused on English (Davison et al, 2019;Bouraoui et al, 2020;Shin et al, 2020;Brown et al, 2020;Alghanmi et al, 2021;, a few notable studies have extended the evaluation to a number of other languages (Jiang et al, 2020;Kassner et al, 2021;Yin et al, 2022). The results of these studies show a large variation in 1 All code and data released at https://github.com/ Betswish/Cross-Lingual-Consistency the extent to which factual knowledge generalizes across languages, revealing yet another facet of language inequality in modern NLP technologies (Hupkes et al, 2022).…”

Interpreting Patient Descriptions using Distantly Supervised Similar Case Retrieval

“…Prior work, however, has shown that existing biomedical LMs often struggle with such tasks. For instance, Alghanmi et al [2] found that the standard BERT model was remarkably competitive with specialised biomedical LMs for inferring diagnoses from patient descriptions. Meng et al [44] furthermore introduced a probing task for evaluating the knowledge captured by biomedical LMs, which also revealed significant issues.…”