Provably Strict Generalisation Benefit for Equivariant Models

Elesedy, Bryn; Zaidi, Sheheryar

doi:10.48550/arxiv.2102.10333

Cited by 9 publications

(40 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While their framework is potentially applicable to general groups, their results only cover the case of translation groups, leading to improvements in sample complexity of at most a factor equal to the dimension. [15] also sudies benefits of group invariance, but focuses on linear models, and only considers interpolating estimators. [10] study benefits of equivariant kernels in structured prediction problems.…”

Section: Related Workmentioning

confidence: 99%

On the Sample Complexity of Learning under Invariance and Geometric Stability

Bietti¹,

Venturi²,

Bruna³

2021

Preprint

View full text Add to dashboard Cite

Many supervised learning problems involve high-dimensional data such as images, text, or graphs. In order to make efficient use of data, it is often useful to leverage certain geometric priors in the problem at hand, such as invariance to translations, permutation subgroups, or stability to small deformations. We study the sample complexity of learning problems where the target function presents such invariance and stability properties, by considering spherical harmonic decompositions of such functions on the sphere. We provide non-parametric rates of convergence for kernel methods, and show improvements in sample complexity by a factor equal to the size of the group when using an invariant kernel over the group, compared to the corresponding non-invariant kernel. These improvements are valid when the sample size is large enough, with an asymptotic behavior that depends on spectral properties of the group. Finally, these gains are extended beyond invariance groups to also cover geometric stability to small deformations, modeled here as subsets (not necessarily subgroups) of permutations.

show abstract

Section: Related Workmentioning

confidence: 99%

On the Sample Complexity of Learning under Invariance and Geometric Stability

Bietti¹,

Venturi²,

Bruna³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Assumptions and technical conditions are given in Section 2 along with an outline of the ideas of Elesedy and Zaidi [8] on which we build. Related works are discussed in Section 5.…”

Section: More Specifically Letmentioning

confidence: 99%

“…However, while implementations and practical applications abound, until very recently a rigorous theoretical justification for invariance was missing. As pointed out in [8], many prior works such as [29,25] provide only worst-case guarantees on the performance of invariant algorithms. It follows that these results do not rule out the possibility of modern training algorithms automatically favouring invariant models, irrespective of the choice of architecture.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Provably Strict Generalisation Benefit for Invariance in Kernel Methods

Elesedy¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

It is a commonly held belief that enforcing invariance improves generalisation. Although this approach enjoys widespread popularity, it is only very recently that a rigorous theoretical demonstration of this benefit has been established. In this work we build on the function space perspective of Elesedy and Zaidi [8] to derive a strictly non-zero generalisation benefit of incorporating invariance in kernel ridge regression when the target is invariant to the action of a compact group. We study invariance enforced by feature averaging and find that generalisation is governed by a notion of effective dimension that arises from the interplay between the kernel and the group. In building towards this result, we find that the action of the group induces an orthogonal decomposition of both the reproducing kernel Hilbert space and its kernel, which may be of interest in its own right.Preprint. Under review.

show abstract

“…In other words, it is not the symmetry of X or Y that is relevant, but the symmetry of the function f mapping from X to Y . If the true relationship in the data has a symmetry, then constraining the hypothesis space to functions f that also have the symmetry makes learning easier and improves generalization [15]. Equivariant models have been developed for a wide variety of symmetries and data types like images [10,58,61,56], sets [60,39], graphs [38], point clouds [3,18,47], dynamical systems [16], jets [6], and other objects [54,17].…”

Section: Representationsmentioning

confidence: 99%

Residual Pathway Priors for Soft Equivariance Constraints

Finzi¹,

Benton²,

Wilson³

2021

Preprint

View full text Add to dashboard Cite

There is often a trade-off between building deep learning systems that are expressive enough to capture the nuances of the reality, and having the right inductive biases for efficient learning. We introduce Residual Pathway Priors (RPPs) as a method for converting hard architectural constraints into soft priors, guiding models towards structured solutions, while retaining the ability to capture additional complexity. Using RPPs, we construct neural network priors with inductive biases for equivariances, but without limiting flexibility. We show that RPPs are resilient to approximate or misspecified symmetries, and are as effective as fully constrained models even when symmetries are exact. We showcase the broad applicability of RPPs with dynamical systems, tabular data, and reinforcement learning. In Mujoco locomotion tasks, where contact forces and directional rewards violate strict equivariance assumptions, the RPP outperforms baseline model-free RL agents, and also improves the learned transition models for model-based RL.

show abstract

Provably Strict Generalisation Benefit for Equivariant Models

Cited by 9 publications

References 17 publications

On the Sample Complexity of Learning under Invariance and Geometric Stability

On the Sample Complexity of Learning under Invariance and Geometric Stability

Provably Strict Generalisation Benefit for Invariance in Kernel Methods

Residual Pathway Priors for Soft Equivariance Constraints

Contact Info

Product

Resources

About