The Representation Theory of Neural Networks

Abstract

In this work, we show that neural networks can be represented via the mathematical theory of quiver representations. More specifically, we prove that a neural network is a quiver representation with activation functions, a mathematical object that we represent using a network quiver. Also, we show that network quivers gently adapt to common neural network concepts such as fully-connected layers, convolution operations, residual connections, batch normalization, and pooling operations. We show that this mathematical representation is by no means an approximation of what neural networks are as it exactly matches reality. This interpretation is algebraic and can be studied with algebraic methods.We also provide a quiver representation model to understand how a neural network creates representations from the data. We show that a neural network saves the data as quiver representations, and maps it to a geometrical space called the moduli space, which is given in terms of the underlying oriented graph of the network. This results as a consequence of our defined objects and of understanding how the neural network computes a prediction in a combinatorial and algebraic way.Overall, representing neural networks through the quiver representation theory leads to 13 consequences that we believe are of great interest to better understand what neural networks are and how they work.

Double framed moduli spaces of quiver representations

“…Definition 7.9. [1] Let Q be a connected finite acyclic quiver without multiple arrows. A neural network over Q is a pair (W, f ) where W is a thin representation of Q and f = (f q ) q∈ Q are activation functions, i.e., almost everywhere differentiable functions f q : C → C.…”

“…where (V, 1) denotes a neural network whose identity activation functions are equal to the identity map (equivalently, activations and pre-activations of vertices are always equal) and (1, ..., 1) ∈ C d Theorem 7.10. [1] Let (W, f ) be a neural network and x ∈ C d an input vector. The following diagram is commutative…”

“…Proof. It follows from the invariance of the network function under the action of G defined in [1] that Ψ is a well-defined functor. For a morphism of neural networks τ : (W, f ) → (V, g) define (1, τ ) : ϕ(W, f ) → ϕ(V, g).…”

“…Moduli spaces of framed quiver representations have been considered since [10,12] and the dual theory (coframed quiver representations) has been also considered independently, see for example [12]. Then, double framing (framing and co-framing at the same time) was found to describe neural networks and the computations they perform on data, see [1]. Motivated by these applications we perform an analysis of the underlying geometric and representation theoretic perspectives of such moduli spaces of double-framed quiver representations.…”

“…We use the representation theoretic approach to these moduli spaces to study the problem from a categorical point of view. Later on, we will present applications to the theory of neural networks as stated in [1].…”