Massif: Interactive Interpretation of Adversarial Attacks on Deep Learning

Abstract

Deep neural networks (DNNs) are increasingly powering high-stakes applications such as autonomous cars and healthcare; however, DNNs are often treated as "black boxes" in such applications. Recent research has also revealed that DNNs are highly vulnerable to adversarial attacks, raising serious concerns over deploying DNNs in the real world. To overcome these deficiencies, we are developing MASSIF, an interactive tool for deciphering adversarial attacks. MASSIF identifies and interactively visualizes neurons and their connections inside a DNN that are strongly activated or suppressed by an adversarial attack. MASSIF provides both a high-level, interpretable overview of the effect of an attack on a DNN, and a low-level, detailed description of the affected neurons. MASSIF's tightly coupled views help people better understand which input features are most vulnerable and important for correct predictions.

NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks

“…For a neuron to detect a feature, it requires an orchestrated interaction among many neurons across different layers. Recent research [10,11,23,41] visually explains how higher level concepts can be constructed by neural connections. In the context of adversarial attack, some methods [10,11,31] identify where in a network the activation pathways of a benign and attacked input instance diverge, and how those diverging activations arrive at an incorrect prediction through connections among neurons.…”

“…Recent research [10,11,23,41] visually explains how higher level concepts can be constructed by neural connections. In the context of adversarial attack, some methods [10,11,31] identify where in a network the activation pathways of a benign and attacked input instance diverge, and how those diverging activations arrive at an incorrect prediction through connections among neurons. Inspired by these techniques, we summarize and visualize how neuron groups interact through connections among them, providing a new way to interpret concept cascading across layers.…”

“…However, such techniques often do not describe how and where the input features are used within the model. Recent research posits a key step towards answering these questions is to interpret neurons (also called channels), since they are highly activated for specific features from the input data [1,10,11,23,42]. While neuron-level interpretation may be a promising approach to discover insights, inspecting individual neurons can be time-consuming; furthermore, individual inspection does not easily reveal how multiple neurons may detect the same features, which means users could easily miss the bigger picture of the DNNs' decision-making processes.…”