Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?

Abstract

Although neural models have achieved impressive results on several NLP benchmarks, little is understood about the mechanisms they use to perform language tasks. Thus, much recent attention has been devoted to analyzing the sentence representations learned by neural encoders, through the lens of 'probing' tasks. However, to what extent was the information encoded in sentence representations, as discovered through a probe, actually used by the model to perform its task? In this work, we examine this probing paradigm through a case study in Natural Language Inference, showing that models can learn to encode linguistic properties even if they are not needed for the task on which the model was trained. We further identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments. Finally, through a set of controlled synthetic tasks, we demonstrate models can encode these properties considerably above chance-level even when distributed in the data as random noise, calling into question the interpretation of absolute claims on probing tasks. 1 * Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.

A Closer Look at How Fine-tuning Changes BERT

“…Ethayarajh, 2019;Mimno and Thompson, 2017), including the recently proposed DIRECTPROBE (Zhou and Srikumar, 2021), which we use in this work. Another line of probing work is to design control tasks (Ravichander et al, 2021;Lan et al, 2020) to reverse-engineer the internal mechanisms of representations (Kovaleva et al, 2019;. However, in contrast to our work, most studies focused on the pre-trained representations, not the fine-tuned ones.…”

On the data requirements of probing

“…Validity measures how well the test measures what it intends to measure. In a valid test, the result is right for the right reasons (McCoy et al, 2019;Ravichander et al, 2021). Robustness measures how well the results of a test can generalize from the experimental setting to realworld settings (Xing et al, 2020;Niu et al, 2020).…”

Diagnosing AI Explanation Methods with Folk Concepts of Behavior

“…This has been a subject of recent criticism for probing methods, on the basis of "correlation does not equal causation", where although probing methods infer that the model represents some concept, no guarantee is given on whether the model actually uses this concept to make its decisions [39,96,112]. This has led to the development of causally-informed class of methods [32,38,115] that do provide a stronger guarantee that causality is correctly attributed, e.g., by showing that the model indeed changes its decision if it ceases to recognize the concept through deriving a counterfactual [28,32].…”