Deep Neural Model Inspection and Comparison via Functional Neuron Pathways

Fiacco, James; Choudhary, Samridhi; Rosé, Carolyn Penstein

doi:10.18653/v1/p19-1575

Cited by 9 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more general approach than investigating linear separability is to analyze the representational similarity between groups of related examples. Previous work used Principal Component Analysis (PCA) [11], Canonical Correlation Analysis (CCA) [31] or clustering techniques [33,23] to find co-activated neurons or to compare representations of different groups of examples. Our introduced technique aims to analyze activations, as well, but in contrast to existing approaches, it allows to compare high-dimensional representations by visual inspection.…”

Section: Data Representation Analysismentioning

confidence: 99%

Visualizing Deep Neural Networks with Topographic Activation Maps

Krug¹,

Ratul²,

Stober³

2022

Preprint

View full text Add to dashboard Cite

Machine Learning with Deep Neural Networks (DNNs) has become a successful tool in solving tasks across various fields of application. The success of DNNs is strongly connected to their high complexity in terms of the number of network layers or of neurons in each layer, which severely complicates to understand how DNNs solve their learned task. To improve the explainability of DNNs, we adapt methods from neuroscience because this field has a rich experience in analyzing complex and opaque systems. In this work, we draw inspiration from how neuroscience uses topographic maps to visualize the activity of the brain when it performs certain tasks. Transferring this approach to DNNs can help to visualize and understand their internal processes more intuitively, too. However, the inner structures of brains and DNNs differ substantially. Therefore, to be able to visualize activations of neurons in DNNs as topographic maps, we research techniques to layout the neurons in a two-dimensional space in which neurons of similar activity are in the vicinity of each other. In this work, we introduce and compare different methods to obtain a topographic layout of the neurons in a network layer. Moreover, we demonstrate how to use the resulting topographic activation maps to identify errors or encoded biases in DNNs or data sets. Our novel visualization technique improves the transparency of DNN-based algorithmic decision-making systems and is accessible to a broad audience because topographic maps are intuitive to interpret without expert-knowledge in Machine Learning.

show abstract

Section: Data Representation Analysismentioning

confidence: 99%

Visualizing Deep Neural Networks with Topographic Activation Maps

Krug¹,

Ratul²,

Stober³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural pathways (Fiacco et al, 2019a) refer to a method for pinpointing sets of a model's neurons that function together in groups. These groups of neurons are referred to as pathways because they cut across architectural layers and allow representation of the flow of activation through a network, potentially from input all the way to output.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

2022

View full text Add to dashboard Cite

As scalable learning technologies become ubiquitous, it generates a large amount of student data, which can be used with machine learning and NLP to develop new instructional technologies, such as personalized practice schedules and adaptive lessons. Additionally, machine learning and NLP are uniquely poised to solve the problems inherent in scaling language instruction to a large number of languages and courses. In this talk, I will describe several projects illustrating these two uses of ML and NLP in language learning at scale at Duolingo -the world's largest language education platform with over 100 courses and around 40 million monthly active learners.

show abstract

“…Such linear classifiers are the basis for the research of Kim et al [40] who derived vectors that represent user-defined concepts. Fiacco et al [41] introduced functional neuron pathways, which are co-activated sets of neurons identified through Principal Component Analysis (PCA). Representational similarity can also be investigated through Canonical Correlation Analysis (CCA) [42] or by clustering of class-specific neuron activations [43].…”

Section: Analyzing Data Set Representationsmentioning

confidence: 99%

Analyzing and Visualizing Deep Neural Networks for Speech Recognition with Saliency-Adjusted Neuron Activation Profiles

et al. 2021

View full text Add to dashboard Cite

Deep Learning-based Automatic Speech Recognition (ASR) models are very successful, but hard to interpret. To gain a better understanding of how Artificial Neural Networks (ANNs) accomplish their tasks, several introspection methods have been proposed. However, established introspection techniques are mostly designed for computer vision tasks and rely on the data being visually interpretable, which limits their usefulness for understanding speech recognition models. To overcome this limitation, we developed a novel neuroscience-inspired technique for visualizing and understanding ANNs, called Saliency-Adjusted Neuron Activation Profiles (SNAPs). SNAPs are a flexible framework to analyze and visualize Deep Neural Networks that does not depend on visually interpretable data. In this work, we demonstrate how to utilize SNAPs for understanding fully-convolutional ASR models. This includes visualizing acoustic concepts learned by the model and the comparative analysis of their representations in the model layers.

show abstract

Deep Neural Model Inspection and Comparison via Functional Neuron Pathways

Cited by 9 publications

References 31 publications

Visualizing Deep Neural Networks with Topographic Activation Maps

Visualizing Deep Neural Networks with Topographic Activation Maps

Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)

Analyzing and Visualizing Deep Neural Networks for Speech Recognition with Saliency-Adjusted Neuron Activation Profiles

Contact Info

Product

Resources

About