Universal approximations of invariant maps by neural networks

Abstract

We describe generalizations of the universal approximation theorem for neural networks to maps invariant or equivariant with respect to linear representations of groups. Our goal is to establish network-like computational models that are both invariant/equivariant and provably complete in the sense of their ability to approximate any continuous invariant/equivariant map. Our contribution is three-fold. First, in the general case of compact groups we propose a construction of a complete invariant/equivariant network using an intermediate polynomial layer. We invoke classical theorems of Hilbert and Weyl to justify and simplify this construction; in particular, we describe an explicit complete ansatz for approximation of permutation-invariant maps. Second, we consider groups of translations and prove several versions of the universal approximation theorem for convolutional networks in the limit of continuous signals on euclidean spaces. Finally, we consider 2D signal transformations equivariant with respect to the group SE(2) of rigid euclidean motions. In this case we introduce the "charge-conserving convnet" -a convnet-like computational model based on the decomposition of the feature space into isotypic representations of SO(2). We prove this model to be a universal approximator for continuous SE(2)-equivariant signal transformations.

Invariant polynomials and machine learning

“…We also require the concept of a (normalised) Haar measure of a compact group H, µ H , which allows us to define an integral over the group, h∈H dµ H . By borrowing the arguments from [7], we can now prove our generalisation as follows.…”

“…But when working with continuous invariant functions, we do not have the luxury of power series representations and so we must resort to other means. In the case of continuous real-valued functions invariant under compact groups, the problem is relatively straightforward and has been solved in [7]. Here, we will prove the generalisation for the case of continuous complex-valued functions invariant under linearly reductive groups defined over C.…”

“…6 Finitely-generated implies that there are finitely many η i 's and free indicates that the η i 's form a linearly independent basis. 7 The primaries are algebraically independent. 8 The secondaries always contain the trivial secondary η 1 = 1.…”

“…Interestingly, unlike real-valued neural networks, the criterion on the activation function for single layer (shallow) and multiple layer (deep) complex-valued networks is different. 1415 In [7], real-valued neural networks were shown to be able to approximate any continuous real-valued function invariant under a compact group using the MAG of the invariant algebra of polynomials as inputs. Here, we will prove the generalisation of that theorem to complex-valued neural networks and continuous complex-valued functions invariant under linearly reductive groups defined over C. 12 A function σ is almost polyharmonic if there exist m ∈ N and an infinitely differentiable g : C → C with ∆ m g = 0 such that σ = g almost everywhere.…”

Scalars are universal: Equivariant machine learning, structured like classical physics

“…Universally approximating invariant functions can be obtained by taking universal non-invariant functions and averaging them over the group orbits [82,58]. However, this approach is not practical for large groups like S n or infinite groups like O(d).…”