BERTnesia: Investigating the capture and forgetting of knowledge in BERT

Abstract

Probing complex language models has recently revealed several insights into linguistic and semantic patterns found in the learned representations. In this article, we probe BERT specifically to understand and measure the relational knowledge it captures in its parametric memory. While probing for linguistic understanding is commonly applied to all layers of BERT as well as finetuned models, this has not been done for factual knowledge. We utilize existing knowledge base completion tasks (LAMA) to probe every layer of pre-trained as well as fine-tuned BERT models (ranking, question answering, NER). Our findings show that knowledge is not just contained in BERT's final layers. Intermediate layers contribute a significant amount (17-60%) to the total knowledge found. Probing intermediate layers also reveals how different types of knowledge emerge at varying rates. When BERT is fine-tuned, relational knowledge is forgotten. The extent of forgetting is impacted by the fine-tuning objective and the training data. We found that ranking models forget the least and retain more knowledge in their final layer compared to masked language modeling and question-answering. However, masked language modeling performed the best at acquiring new knowledge from the training data. When it comes to learning facts, we found that capacity and fact density are key factors. We hope this initial work will spur further research into understanding the parametric memory of language models and the effect of training objectives on factual knowledge. The code to repeat the experiments is publicly available on GitHub 1 .

Understanding Finetuning for Factual Knowledge Extraction from Language Models

“…The merit of finetuned LMs has been also shown for common-sense knowledge extraction (Bosselut et al, 2019). Previous work also studies the effect of dataset size for finetuning (Wallat et al, 2021;Fichtel et al, 2021;Da et al, 2021), but the negative effects finetuning (studied in this paper) remain unexplored. For a full review of the literature on knowledge probing and extraction, we refer to (Safavi & Koutra, 2021;AlKhamissi et al, 2022).…”

“…While previous work typically explains the phenomanon in Figure 1 as forgetting effect (Wallat et al, 2021), our study reveals a more nuanced explanation in terms of Frequency Shock: even though both "Moscow" and "Baku" have been observed an equal number of times in the training set, since "Baku" is expected to be a less common entity 2 and hence less observed during the pre-training of the language model, the finetuned model receives a frequency shock leading to an over-prediction of the entity "Baku", hence corrupting an originally correct prediction. Note that Frequency Shock and Range Shift are related to the problem of out-of-distribution (OOD) generalization in machine learning, see section 3.6 for more discussion.…”

“…Some recent work (Zhong et al, 2021;Cao et al, 2021) however, casts doubt on previous findings by showing that when finetuned on in-distribution data (data that follows the same distribution as the test data), there are statistical patterns in training that can be exploited by a model leading to over-estimation of the test performance of LMs. Moreover, Wallat et al (2021) show that finetuning may lead to forgetting the previously known facts by the model. Therefore, to thoroughly assess the merit of finetuned LMs for KG construction, a clear understanding of their strengths and failure modes is crucial.…”

A Review on Language Models as Knowledge Bases

“…There are increasing evidence that show that scaling LMs to larger sizes is not the solution to generating factually correct information (Lazaridou et al, 2021;Gehman et al, 2020;Lin et al, 2021a). As a result, this would also result in catastrophic forgetting (Wallat et al, 2021). Changing a single weight may have a ripple effect that affects a large number of other implicitly memorized facts.…”

“…commonsense question answering) so it can make way for the required knowledge to surface in the output during evaluation. Previous work has shown that most knowledge encoded in a LM are acquired during pretraining, while finetuning just learns an interface to access that acquired knowledge (Da et al, 2021;Wallat et al, 2021).…”