NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks

Park, Haekyu; Das, Nilaksh; Duggal, Rahul; Wright, Austin P.; Shaikh, Omar; Hohman, Fred; Chau, Duen Horng

doi:10.1109/tvcg.2021.3114858

Cited by 16 publications

(22 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inspired by TCAV, Neuron Shapley [16] and ACE [15] automatically find and quantify concepts detected by a model. NeuroCartography [35] summarizes concepts detected by neurons and visualizes the relationship of concepts by computing vector representations of neurons based on their conceptual neighborhood. However, the embedding spaces generated by those previous works are dedicated only to a single model.…”

Section: Interpreting Dnns After Trainingmentioning

confidence: 99%

“…Inspired by the intuition that neurons detecting similar concepts are activated by many common input images [35], we encode concepts of neurons co-activated by more common images to be closer on the semantic space. We use the neuron embedding method proposed in [35] but add detailed problem formulation for encoding neurons' concepts in Appendix A, which is not clearly stated in the previous work.…”

Section: 21mentioning

confidence: 99%

“…To find stimuli for each neuron, we decide to measure the activation strength of a neuron with the maximum value of a neuron's activation map (also called feature maps of a channel). This is an indicator of how strongly a concept is detected by the neuron, as used in [35]. For each neuron, we create a neuron's stimuli by collecting k images that lead to the highest maximum of the neuron's activation map given those images.…”

Section: 21mentioning

confidence: 99%

“…Thus, our first experiment aims to verify whether the concepts detected by Con-ceptEvo remain semantically consistent and aligned in the embedding space across models and training epochs. To this end, we conducted a large-scale human evaluation using Amazon Mechanical Turk (MTurk) modeled on previous works measuring conceptual consistency in computer vision tasks [15,35], and we developed four groups of concepts to evaluate. We provided 260 unique MTurk participants with nine concept classification tasks.…”

Section: Alignment Of Neuron Embeddingsmentioning

confidence: 99%

“…For example, saliencybased approaches find what regions of an image are being detected for a prediction [43,45]. Conceptbased methods discover concepts learned by a model (e.g., "pointy ears" concepts detected by a model in Figure 1) and how those concepts interact to form higher-level concepts and predictions [35,32,15,21,3].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

ConceptEvo: Interpreting Concept Evolution in Deep Learning Training

Park¹,

Lee²,

Hoover³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) have been widely used for decision making, prompting a surge of interest in interpreting how these complex models work. Recent literature on DNN interpretation has revolved around already-trained models; however, much less research focuses on interpreting how the models evolve as they are trained. Interpreting model evolution is crucial to monitor network training and can aid proactive decisions about necessary interventions. In this work, we present ConceptEvo, a general interpretation framework for DNNs that reveals the inception and evolution of detected concepts during training. Through a large-scale human evaluation with 260 participants and quantitative experiments, we show that ConceptEvo discovers evolution across different models that are meaningful to humans, helpful for early-training intervention decisions, and crucial to the prediction for a given class.

show abstract

Section: Interpreting Dnns After Trainingmentioning

confidence: 99%

Section: 21mentioning

confidence: 99%

Section: 21mentioning

confidence: 99%

Section: Alignment Of Neuron Embeddingsmentioning

confidence: 99%