Predictor networks and stop-grads provide implicit variance regularization in BYOL/SimSiam

Halvagal, Manu Srinath; Laborieux, Axel; Zenke, Friedemann

doi:10.48550/arxiv.2212.04858

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Locally instructive cues may arise from the supervisory action of other brain areas, as assumed in computational models of error-driven learning 55,56,57 . Alternatively, locally instructive cues may be features of the late-stage dynamics of response, as assumed in models of self-supervised learning 58 . Future work will be required to formally address these possibilities, and to gain a mechanistic understanding of the behavioral manifestations of BTSP in the PFC.…”

Section: Discussionmentioning

confidence: 99%

Cellular Substrate of Eligibility Traces

Caya-Bissonnette

Naud

Béı̈que

2023

Preprint

View full text Add to dashboard Cite

The ability of synapses to undergo associative, activity-dependent weight changes constitutes a linchpin of current cellular models of learning and memory. It is, however, unclear whether canonical forms of Hebbian plasticity, which inherently detect correlations of cellular events occurring over short time scales, can solve the temporal credit assignment problem proper to learning driven by delayed behavioral outcomes. Recent evidence supports the existence of synaptic eligibility traces, a time decaying process that renders synapses momentarily eligible for a weight update by a delayed instructive signal. While eligibility traces offer a means of retrospective credit assignment, their material nature is unknown. Here, we combined whole-cell recordings with two-photon uncaging, calcium imaging and biophysical modeling to address this question. We observed and parameterized a form of behavioral timescale synaptic plasticity (BTSP) in layer 5 pyramidal neurons of mice prefrontal areas wherein the pairing of temporally separated pre- and postsynaptic events (0.5 s to 1 s), irrespective of order, induced synaptic potentiation. By imaging calcium in apical oblique dendrites, we reveal a short-term and associative plasticity of calcium dynamics (STAPCD) whose time-dependence mirrored the induction rules of BTSP. We identified a core set of molecular players that were essential for both STAPCD and BTSP and that, together with computational simulations, support a model wherein the dynamics of intracellular handling of calcium by the endoplasmic reticulum (ER) provides a latent memory trace of neural activity that instantiates synaptic weight updates upon a delayed instructive signal. By satisfying the requirements expected of eligibility traces, this mechanism accounts for how individual neurons can conjunctively bind cellular events that are separated by behaviorally relevant temporal delays, and thus offers a cellular model of reinforced learning.

show abstract

Section: Discussionmentioning

confidence: 99%

Cellular Substrate of Eligibility Traces

Caya-Bissonnette

Naud

Béı̈que

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Its structure is similar to BYOL, retaining the predictor of the online network, but without EMA (exponential moving average). Simsiam proves that EMA is not necessary to prevent collapse but removing it will sacrifice part of the accuracy [32].…”

Section: Contrastive Learningmentioning

confidence: 99%

Unlocking the Potential of Data Augmentation in Contrastive Learning for Hyperspectral Image Classification

Yan

2023

Remote Sensing

View full text Add to dashboard Cite

Despite the rapid development of deep learning in hyperspectral image classification (HSIC), most models require a large amount of labeled data, which are both time-consuming and laborious to obtain. However, contrastive learning can extract spatial–spectral features from samples without labels, which helps to solve the above problem. Our focus is on optimizing the contrastive learning process and improving feature extraction from all samples. In this study, we propose the Unlocking-the-Potential-of-Data-Augmentation (UPDA) strategy, which involves adding superior data augmentation methods to enhance the representation of features extracted by contrastive learning. Specifically, we introduce three augmentation methods—band erasure, gradient mask, and random occlusion—to the Bootstrap-Your-Own-Latent (BYOL) structure. Our experimental results demonstrate that our method can effectively improve feature representation and thus improve classification accuracy. Additionally, we conduct ablation experiments to explore the effectiveness of different data augmentation methods.

show abstract

Predictor networks and stop-grads provide implicit variance regularization in BYOL/SimSiam

Cited by 2 publications

References 0 publications

Cellular Substrate of Eligibility Traces

Cellular Substrate of Eligibility Traces

Unlocking the Potential of Data Augmentation in Contrastive Learning for Hyperspectral Image Classification

Contact Info

Product

Resources

About