Locating and Editing Factual Associations in GPT

Meng, Kevin; Bau, David; Andonian, Alex; Belinkov, Yonatan

doi:10.48550/arxiv.2202.05262

Cited by 21 publications

(27 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, a surge of works have investigated the knowledge captured by the FFN layers (Da et al, 2021;Dai et al, 2021;Yao et al, 2022;Meng et al, 2022;Wallat et al, 2020). These works show that the FFN layers store various types of knowledge, which can be located in specific neurons and edited.…”

Section: Related Workmentioning

confidence: 99%

“…We study this question through the lens of the feed-forward network (FFN) layers, one of the core components in transformers (Vaswani et al, 2017). Recent work showed that these layers play an important role in LMs, acting as memories that encode factual and linguistic knowledge (Geva et al, 2021;Da et al, 2021;Meng et al, 2022). In this work, we investigate how outputs from the FFN layers are utilized internally to build predictions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

Geva¹,

Caciularu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer-based language models (LMs) are at the core of modern NLP, but their internal prediction construction process is opaque and largely not understood. In this work, we make a substantial step towards unveiling this underlying prediction process, by reverseengineering the operation of the feed-forward network (FFN) layers, one of the building blocks of transformer models. We view the token representation as a changing distribution over the vocabulary, and the output from each FFN layer as an additive update to that distribution. Then, we analyze the FFN updates in the vocabulary space, showing that each update can be decomposed to sub-updates corresponding to single FFN parameter vectors, each promoting concepts that are often humaninterpretable. We then leverage these findings for controlling LM predictions, where we reduce the toxicity of GPT2 by almost 50%, and for improving computation efficiency with a simple early exit rule, saving 20% of computation on average. 1 * Equal contribution. † Work done during an internship at AI2. 1 https://github.com/aviclu/ffn-values.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

Geva¹,

Caciularu²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Transformer Mechanisms We evaluate transformers and whether they are explainable via other mechanisms, such as the feedfoward layers Geva et al, 2021;Meng et al, 2022). propose a tool to measure nonlinearities in LMs by taking into account geometry space of embeddings, finding that the non-linearity of the self-attention feedforward layers and MLPs of a LM follow similar patterns, but their functions are less known.…”

Section: Interpretabilitymentioning

confidence: 99%

“…propose a tool to measure nonlinearities in LMs by taking into account geometry space of embeddings, finding that the non-linearity of the self-attention feedforward layers and MLPs of a LM follow similar patterns, but their functions are less known. Geva et al (2021) extrapolate from this learned fact and find that feedforward layers in LMs are just key value memories; as a result Meng et al (2022) are able to use their method of causal tracing to locate the knowledge and use the key value pairs to access knowledge within the feedforward layers and make modifications to it.…”

Section: Interpretabilitymentioning

confidence: 99%

A Review on Language Models as Knowledge Bases

Badr¹,

Li²,

Çelikyılmaz³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem an LM should have to fully act as a KB, and review the recent literature with respect to those aspects. 1

show abstract

“…Our intervention in FFN sub-updates relates to recent methods for locating and editing knowledge in the FFN layers of LMs (Meng et al, 2022;Dai et al, 2021). Different from these methods, LM-Debugger aims to provide a comprehensive and fine-grained interpretation of the prediction construction process across the layers.…”

Section: Related Workmentioning

confidence: 99%

LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models

Geva¹,

Caciularu²,

Dar³

et al. 2022

Preprint

View full text Add to dashboard Cite

The opaque nature and unexplained behavior of transformer-based language models (LMs) have spurred a wide interest in interpreting their predictions. However, current interpretation methods mostly focus on probing models from outside, executing behavioral tests, and analyzing salience input features, while the internal prediction construction process is largely not understood. In this work, we introduce LM-Debugger, an interactive debugger tool for transformer-based LMs, which provides a fine-grained interpretation of the model's internal prediction process, as well as a powerful framework for intervening in LM behavior. For its backbone, LM-Debugger relies on a recent method that interprets the inner token representations and their updates by the feed-forward layers in the vocabulary space. We demonstrate the utility of LM-Debugger for single-prediction debugging, by inspecting the internal disambiguation process done by GPT2. Moreover, we show how easily LM-Debugger allows to shift model behavior in a direction of the user's choice, by identifying a few vectors in the network and inducing effective interventions to the prediction process. We release LM-Debugger as an open-source tool and a demo over GPT2 models.

show abstract

Locating and Editing Factual Associations in GPT

Cited by 21 publications

References 13 publications

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

A Review on Language Models as Knowledge Bases

LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models

Contact Info

Product

Resources

About