Provably Strict Generalisation Benefit for Invariance in Kernel Methods

Abstract

It is a commonly held belief that enforcing invariance improves generalisation. Although this approach enjoys widespread popularity, it is only very recently that a rigorous theoretical demonstration of this benefit has been established. In this work we build on the function space perspective of Elesedy and Zaidi [8] to derive a strictly non-zero generalisation benefit of incorporating invariance in kernel ridge regression when the target is invariant to the action of a compact group. We study invariance enforced by feature averaging and find that generalisation is governed by a notion of effective dimension that arises from the interplay between the kernel and the group. In building towards this result, we find that the action of the group induces an orthogonal decomposition of both the reproducing kernel Hilbert space and its kernel, which may be of interest in its own right.Preprint. Under review.

Dimensionless machine learning: Imposing exact units equivariance

“…Recent results show how to compute explicitly the gains in terms of generalization gaps and sample complexity of imposing group invariances and equivariances in machine learning. The results from Mei et al (2021) and Bietti et al (2021) hold for finite groups, whereas the results from Elesedy and Zaidi (2021) and Elesedy (2021) hold for general compact groups. Specifically, Elesedy and Zaidi (2021) shows that if we are aiming to learn a target invariant function f * from samples from an invariant distribution µ, then for any estimator f , the projection of f onto the space of invariant functions has smaller expected error.…”

“…A recent line of work develops mathematical theory to quantify the benefits of imposing symmetries and group equivariance in regression problems. The work of Elesedy and Zaidi (2021) provides a computation of the excess risk in the context of linear regression, then extended to kernel regressions in Elesedy (2021). The works of Bietti et al (2021) and Mei et al (2021) focus on quantifying the sample complexity gain for regressions with (finite) group-invariant kernels.…”

“…A slightly more general formulation allows them to express the same ideas for equivariance. This work has been extended to kernel regressions (Elesedy, 2021).…”