The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods

Thiry, Louis; Arbel, Michael; Belilovsky, Eugene; Oyallon, Edouard

doi:10.48550/arxiv.2101.07528

Cited by 3 publications

(2 citation statements)

References 27 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar convergences are observed for auditory processing, as activity of CNN trained on auditory tasks can predict neural activity in the corresponding cortices [59], or for networks that predict fMRI and MEG responses during language processing tasks [60]. Cutting-edge developments in artificial visual systems incorporate ideas from natural language processing such as transformers or local context embedding [61,62]. Researchers are still struggling to understand precisely how these mechanisms operate, or why they achieve such high performances.…”

Section: Hierarchical Processing Of Sensory Inputsmentioning

confidence: 71%

Evolution of Brains and Computers: The Roads Not Taken

Solé

Seoane

2022

Entropy

View full text Add to dashboard Cite

When computers started to become a dominant part of technology around the 1950s, fundamental questions about reliable designs and robustness were of great relevance. Their development gave rise to the exploration of new questions, such as what made brains reliable (since neurons can die) and how computers could get inspiration from neural systems. In parallel, the first artificial neural networks came to life. Since then, the comparative view between brains and computers has been developed in new, sometimes unexpected directions. With the rise of deep learning and the development of connectomics, an evolutionary look at how both hardware and neural complexity have evolved or designed is required. In this paper, we argue that important similarities have resulted both from convergent evolution (the inevitable outcome of architectural constraints) and inspiration of hardware and software principles guided by toy pictures of neurobiology. Moreover, dissimilarities and gaps originate from the lack of major innovations that have paved the way to biological computing (including brains) that are completely absent within the artificial domain. As it occurs within synthetic biocomputation, we can also ask whether alternative minds can emerge from A.I. designs. Here, we take an evolutionary view of the problem and discuss the remarkable convergences between living and artificial designs and what are the pre-conditions to achieve artificial intelligence.

show abstract

Section: Hierarchical Processing Of Sensory Inputsmentioning

confidence: 71%

Evolution of Brains and Computers: The Roads Not Taken

Solé

Seoane

2022

Entropy

View full text Add to dashboard Cite

show abstract

“…Many works have explored the effectiveness of path-based image features. In the supervised setting, Bagnet [Brendel and Bethge, 2018] and Thiry et al [2021] showed that aggregation of patch-based features can achieve most of the performance of supervised learning on Image datasets. In the unsupervised setting, Gidaris et al [2020] performs SSL by requiring a bag-of patches representation to be invariant between different views.…”

Section: Patch-based Representationmentioning

confidence: 99%

Intra-Instance VICReg: Bag of Self-Supervised Image Patch Embedding

Chen¹,

Bardes²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, self-supervised learning (SSL) has achieved tremendous empirical advancements in learning image representation. However, our understanding and knowledge of the representation are still limited. This work shows that the success of the SOTA siamese-network-based SSL approaches is primarily based on learning a representation of image patches. Particularly, we show that when we learn a representation only for fixed-scale image patches and aggregate different patch representations linearly for an image (instance), it can achieve on par or even better results than the baseline methods on several benchmarks. Further, we show that the patch representation aggregation can also improve various SOTA baseline methods by a large margin. We also establish a formal connection between the SSL objective and the image patches co-occurrence statistics modeling, which supplements the prevailing invariance perspective. By visualizing the nearest neighbors of different image patches in the embedding space and projection space, we show that while the projection has more invariance, the embedding space tends to preserve more equivariance and locality. Finally, we propose a hypothesis for the future direction based on the discovery of this work.Preprint. Under review.

show abstract

Evolution of Brains and Computers: The Roads not Taken

Solé¹

2022

Preprint

View full text Add to dashboard Cite

When computers start to become a dominant part of technology around the 1950s, fundamental questions about reliable designs and robustness were of great relevance. Their development gave rise to the exploration of new questions such as what made brains reliable (since neurons can die) and how computers could get inspiration from neural systems. In parallel, the first Artificial Neural Networks came to life. Since then, the comparative view between brains and computers has been developed in new, sometimes unsuspected directions. With the rise of deep learning and the development of connectomics, an evolutionary look at how both hardware and neural complexity have evolved or designed is required. In this paper, we argue that important similarities have resulted both from convergent evolution (the inevitable outcome of architectural constraints) and inspiration of hardware and software principles guided by toy pictures of neurobiology. Moreover, dissimilarities and gaps originate from the lack of major innovations that have paved the way to biological computing (including brains) that are completely absent within the artificial domain. As it occurs within synthetic biocomputation, we can also ask whether alternative minds can emerge from A.I.\ designs. Here we take an evolutionary view of the problem and discuss the remarkable convergences between living and artificial designs and what are the pre-conditions to achieve artificial intelligence.

show abstract

The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods

Cited by 3 publications

References 27 publications

Evolution of Brains and Computers: The Roads Not Taken

Evolution of Brains and Computers: The Roads Not Taken

Intra-Instance VICReg: Bag of Self-Supervised Image Patch Embedding

Evolution of Brains and Computers: The Roads not Taken

Contact Info

Product

Resources

About