Neural network interpretation using descrambler groups

Abstract

The lack of interpretability and trust is a much-criticized feature of deep neural networks. In fully connected nets, the signaling between inner layers is scrambled because backpropagation training does not require perceptrons to be arranged in any particular order. The result is a black box; this problem is particularly severe in scientific computing and digital signal processing (DSP), where neural nets perform abstract mathematical transformations that do not reduce to features or concepts. We present here a group-theoretical procedure that attempts to bring inner-layer signaling into a human-readable form, the assumption being that this form exists and has identifiable and quantifiable features—for example, smoothness or locality. We applied the proposed method to DEERNet (a DSP network used in electron spin resonance) and managed to descramble it. We found considerable internal sophistication: the network spontaneously invents a bandpass filter, a notch filter, a frequency axis rescaling transformation, frequency-division multiplexing, group embedding, spectral filtering regularization, and a map from harmonic functions into Chebyshev polynomials—in 10 min of unattended training from a random initial guess.

FID-Net: A versatile deep neural network architecture for NMR spectral reconstruction and virtual decoupling

“…In a very recent and elegant study, it was shown that a neural network could be interpreted and thus it was shown how each of the hidden layers can be mapped to specific actions and mathematical transformations (Amey et al 2021 ). Such an approach is naturally highly attractive in order to fully understand the strengths and weaknesses of a DNN.…”

FID-Net: A Versatile Deep Neural Network Architecture for NMR Spectral Reconstruction and Virtual Decoupling

“…In a very recent and elegant study, it was shown that a neural network could be interpretated and thus it was shown how each of the hidden layers can be mapped to specific actions and mathematical transformations 18 . Such an approach is naturally highly attractive in order to fully understand the strengths and weaknesses of a DNN.…”

Is There a Need for a More Precise Description of Biomolecule Interactions to Understand Cell Function?

“…However, despite impressive successes met by artificial intelligence in finding patterns in complex datasets and predicting outputs, some unanswered questions hamper their contribution to our understanding of cell function. First, the complexity of trained networks caused them to be considered as black boxes, and attempts are currently done to decrease their opacity [54,55] or increase reliability [56,57]. Secondly, a well-known caveat of correlation studies is that a strong correlation between two parameters is not a proof of a causal relationship between these, and unveiling causal relationships is by no means a simple task [58].…”