What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

Abstract

Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network's performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neurons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available 1 as part of the NeuroX toolkit (Dalvi et al. 2019a).

On the Linguistic Representational Power of Neural Machine Translation Models

“…They found the language model representations to consistently outperform those from NMT. In other work, we have found that language model representations are of similar quality to NMT ones in terms of POS and morphology, but are behind in terms of semantic tagging (Dalvi et al 2019a). Tenney et al (2019) compared representations from CoVE, ELMo, GPT, and BERT on a number of classiciation tasks, partially overlapping with the ones we study.…”

“…We analyzed individual tags across layers and found that open class categories such as verbs and nouns are distributed across several layers, although the majority of the learning of these phenomena is still done at layer 1. Please refer to (Dalvi et al 2019a) for further information. Figure 6: Morphological tagging accuracy using representations from layers 1 to 4, taken from encoders and decoders of different language pairs.…”

On the Realization of Compositionality in Neural Networks

“…For example, if the input was '000 t1 t2', we trained the classifier to predict 't1' for the encoder activations of the second time step and to predict 't2' for the activations of the third time step. Similarly to the methodology of Dalvi et al (2019), we subsequently added units to a set, depending on the absolute weight they were assigned in the diagnostic classifier. 4 After each addition, we re-calculated the accuracy for the prediction.…”

“…However, unlikeDalvi et al (2019), we do not use any regularization on the DC to contrast the different degrees to which information is distributed across neurons in the two model types.5 Applying the methods development by Lundberg and Lee (2017) seems to confirm the responsible neurons we found, but selects more neurons and gives less consistent results, which we trace back to the extensive approximations required and some model assumptions (e.g. feature independence) being violated.…”

NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks

“…We provide three methods to analyze neural network models in the toolkit: Individual and Cross-model Analysis, to search for neurons, important for the model itself (Bau et al 2019) and Linguistic Correlation Analysis, which identifies important neurons w.r.t. an extrinsic property (Dalvi et al 2019). The output of each method is a ranked list of neurons.…”