Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers

Abstract

Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks.

Geometric and Topological Inference for Deep Representations of Complex Networks

“…The entry point is [16], where we propose a perspective to understand neural network optimization as a partially observable model selection problem. In our subsequent work [18], we introduce the details of how to approximate the minimum description length (MDL) between neural network layers and demonstrate that using MDL as the regularity information is useful, from an engineering angle, for neural networks to learn from certain input data distributions. By comparing with other theoretical tools such as mutual information, we envision it as a tool to understand how information propagated between network modules, a venue not widely explored previously, as in [17,18].…”

“…We introduce higher-order simplicial structure as a new summary statistic, and discover that these networks contain an abundance of cliques of single-cell profiles bound into cavities that guide the emergence of more complicated habitation forms [15,19]. To further disentangle information flow, we develop a mathematical filtration technique to compute nerve balls in a dual metric of space and time [20] and a information-theoretical measure among network modules [16,18]. This work aims to solve the following two analytical problems.…”

A Short Review on Minimum Description Length: An Application to Dimension Reduction in PCA

“…In particular, it has been used for ensuring that there is less information in the weights than in the output vectors of the training cases; to this aim, the model cost is the number of bits it takes to describe the weights, and the cost of the data given the model is the number of bits it takes to describe the discrepancy between the correct output and the output of the neural network on each training case. Very recently, in [ 42 ], the neural network training process has been seen as a model selection problem, and the model complexity of its layers has been computed as the optimal universal code length by means of a normalized maximum likelihood formulation. This kind of approach offers a new tool for analyzing and understanding neural networks while speeding up the training phase and increasing the sensitivity to imbalanced data.…”