Neurashed: A Phenomenological Model for Imitating Deep Learning Training

Abstract

To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a hierarchically structured network architecture, parameters iteratively optimized using stochastic gradient-based methods, and information from the data that evolves compressively. As an instantiation, we integrate these characteristics into a graphical model called neurashed. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories.

A Law of Data Separation in Deep Learning

“…Two-layer block Three-layer block Mixed Because each module makes equal but small contributions, a few neurons in a single layer are unlikely to explain how a deep learning model makes a particular prediction. It is therefore crucial to take all layers collectively for interpretation [32]. This view, however, challenges the layer-wise approaches to deep learning interpretation [38,34].…”