Stress Test Evaluation of Biomedical Word Embeddings

Abstract

The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe "stress" scenarios. In this work, we systematically evaluate three language models with adversarial examples -automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.

Augmenting physicians with artificial intelligence to transform healthcare: Challenges and opportunities

“…Goodfellow et al (2015) found that when small but intentionally worst‐case perturbations are applied to the input to generate “adversarial examples,” the model can output an incorrect answer with high confidence. Such scenarios are also prevalent in medical AI models in both medical imaging tasks and NLP tasks (Araujo et al, 2020; Ozbulak et al, 2019). In this case, it is difficult to gauge AI's frontier for generalizability and robustness.…”

Evaluation Benchmarks for Spanish Sentence Representations

“…Following this approach, SentEval (Conneau and Kiela, 2018) and DiscoEval (Chen et al, 2019) include tasks at the sentence and discourse level. The third focuses on stress tests (Naik et al, 2018;Aspillaga et al, 2020;Araujo et al, 2021a) that seek to assess the ability of language models to adapt to cases designed to confuse them. The fourth objective is an evaluation from a linguistic perspective (Warstadt et al, 2019;Ettinger, 2020;Puccetti et al, 2021) to elucidate the models' actual linguistic capacities or knowledge.…”