Re-basin via implicit Sinkhorn differentiation

Peña, Fidel A. Guerrero; Medeiros, Heitor R.; Dubail, Thomas; Aminbeidokhti, Masih; Granger, Éric; Pedersoli, Marco

doi:10.48550/arxiv.2212.12042

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The symmetries described in Equation (4) were used in several studies in the last few years, mainly to investigate the loss landscape of neural networks (Brea et al, 2019;Tatro et al, 2020;Arjevani & Field, 2021;Entezari et al, 2021;Simsek et al, 2021;Ainsworth et al, 2022;Peña et al, 2022), but also in Schürholt et al (2021) as a motivation for a data augmentation scheme.…”

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%

Equivariant Architectures for Learning in Deep Weight Spaces

Navon¹,

Shamsian²,

Achituve³

et al. 2023

Preprint

View full text Add to dashboard Cite

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

show abstract

Section: Permutation Symmetries Of Neural Networkmentioning

confidence: 99%