On the Sample Complexity of Learning under Invariance and Geometric Stability

Bietti, Alberto; Venturi, Luca; Bruna, Joan

doi:10.48550/arxiv.2106.07148

Cited by 3 publications

(3 citation statements)

References 13 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples include fluid dynamics [20,24,29,51], molecular dynamics [1,43,58], quantum mechanics [25,26,48], etc. Theoretical studies have also been conducted to show the benefit of preserving symmetry during learning [3,12,21,30].…”

Section: Background and Related Work 21 Neural Network And Symmetriesmentioning

confidence: 99%

Why Self-Attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

null,

Ying

2023

JML

View full text Add to dashboard Cite

In this paper, we show that structures similar to self-attention are natural for learning many sequence-to-sequence problems from the perspective of symmetry. Inspired by language processing applications, we study the orthogonal equivariance of seq2seq functions with knowledge, which are functions taking two inputs -an input sequence and a knowledge -and outputting another sequence. The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence. We show that orthogonal equivariance in the embedding space is natural for seq2seq functions with knowledge, and under such equivariance, the function must take a form close to self-attention. This shows that network structures similar to self-attention are the right structures for representing the target function of many seq2seq problems. The representation can be further refined if a finite information principle is considered, or a permutation equivariance holds for the elements of the input sequence.

show abstract

Section: Background and Related Work 21 Neural Network And Symmetriesmentioning

confidence: 99%

Why Self-Attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

null,

Ying

2023

JML

View full text Add to dashboard Cite

show abstract

“…Such ideas can be generalized in terms of symmetries in the world [93]. Machine learning has developed many techniques to incorporate known invariances to the learning process [94][95][96][97], as well as mathematically quantifying how much one can gain by imposing them [98,99]. In fact, in many cases we may want to think about constraints themselves as something to be learned [100,101], a process that would unfold over evolutionary timescales for NIs.…”

Section: Constraintsmentioning

confidence: 99%

Prospective Learning: Back to the Future

Vogelstein¹,

Verstynen²,

Kording³

et al. 2022

Preprint

View full text Add to dashboard Cite

Research on both natural intelligence (NI) and artificial intelligence (AI) generally assumes that the future resembles the past: intelligent agents or systems (what we call 'intelligence') observe and act on the world, then use this experience to act on future experiences of the same kind. We call this 'retrospective learning'. For example, an intelligence may see a set of pictures of objects, along with their names, and learn to name them. A retrospective learning intelligence would merely be able to name more pictures of the same objects. We argue that this is not what true intelligence is about. In many real world problems, both NIs and AIs will have to learn for an uncertain future. Both must update their internal models to be useful for future tasks, such as naming fundamentally new objects and using these objects effectively in a new context or to achieve previously unencountered goals. This ability to learn for the future we call 'prospective learning'. We articulate four relevant factors that jointly define prospective learning. Continual learning enables intelligences to remember those aspects of the past which it believes will be most useful in the future. Prospective constraints (including biases and priors) facilitate the intelligence finding general solutions that will be applicable to future problems. Curiosity motivates taking actions that inform future decision making, including in previously unmet situations. Causal estimation enables learning the structure of relations that guide choosing actions for specific outcomes, even when the specific action-outcome contingencies have never been observed before. We argue that a paradigm shift from retrospective to prospective learning will enable the communities that study intelligence to unite and overcome existing bottlenecks to more effectively explain, augment, and engineer intelligences."No man ever steps in the same river twice. For it's not the same river and he's not the same man." -Heraclitus

show abstract

“…Their work focuses on the benefits of group invariance in reducing the sample size. [10] considered general kernels that incorporate group invariance and derived a generalization bound based on counting the number of eigenfunctions of the kernel. [28,27] proposed a convolutional kernel network (CKN) that involves layers of patch extraction, convolution, and pooling.…”

Section: Related Workmentioning

confidence: 99%

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

Geifman¹,

Galun²,

Jacobs³

2022

Preprint

View full text Add to dashboard Cite

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical facotorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.

show abstract

On the Sample Complexity of Learning under Invariance and Geometric Stability

Cited by 3 publications

References 13 publications

Why Self-Attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

Why Self-Attention is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

Prospective Learning: Back to the Future

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

Contact Info

Product

Resources

About