Multitasking Associative Networks

Agliari, Elena; Barra, Adriano; Galluzzi, Andrea; Guerra, Francesco; Moauro, Francesco

doi:10.1103/physrevlett.109.268101

Cited by 110 publications

(167 citation statements)

References 17 publications

Supporting

Mentioning

160

Contrasting

Order By: Relevance

“…2) provided that nonlinear Φ (ReLU) and appropriate threshold values θ are considered. The presence of the fields g i acting on the visible units (absent in the v i = ±1 model of [17][18][19]), is also crucial for the existence of our compositional phase as explained above.It would be interesting to extend our work to more than one layers of hidden units, or to other types of nonlinear Φ. While numerical studies of RBMs with Bernoulli hidden units show no qualitative change compared to ReLU, choosing Φ(h) growing asymptotically faster than h could affect the nature of the extracted features [23].…”

mentioning

confidence: 99%

mentioning

confidence: 99%

“…In this framework magnetized hidden units identify retrieved patterns, and α corresponds to the capacity of the autoassociative memory. Tsodyks and Feigel'man showed how the critical capacity (for single pattern retrieval) could be dramatically increased with sparse weights (p 1) and appropriate tuning of the fields g i [21]; however this effect could be achieved only with vanishingly low activities q. Agliari and collaborators showed in a series of papers [17,18] that multiple sparse patterns could be simultaneously retrieved in the case of linear Φ and vanishing capacity α = 0 (finite M ). Finite capacity α ∼ c −2 could be achieved at zero 5 temperature in the limit of extreme sparsity, p = c/N , only [19]; for MNIST p 0.1 and N = 784 would give α ∼ 2.10 −4 .…”

mentioning

confidence: 99%

“…2(d)) measures the fraction of nonzero weights, p = i p i /N . This distribution was previously introduced to study parallel storage of multiple sparse items in the Hopfield model [17,18]. For simplicity the fields on visible units and the potentials acting on hidden units are chosen to be uniform, g i = g and U µ = U ReLU ( Fig.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Emergence of Compositional Representations in Restricted Boltzmann Machines

2017

View full text Add to dashboard Cite

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine-learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST.Recent years have witnessed major progresses in supervised machine learning, e.g. in video, audio, image processing,... [1]. Despite those impressive successes, unsupervised learning, in which the structure of data is learned without a priori knowledge still presents formidable challenges. A fundamental question is how to learn probability distributions that fit well complex data manifolds in high-dimensional spaces [2]. Once learnt, such generative models can be used for denoising, completion, artificial data generation,... Hereafter we focus on one important generative model, Restricted Boltzmann Machines (RBM) [3, 4]. In its simplest formulation a RBM is a Boltzmann machine on a bipartite graph, see Fig. 1(a), with a visible (v) layer that represents the data, connected to a hidden (h) layer meant to extract and explain their statistical features. The marginal distribution over the visible layer is fitted to the data through approximate likelihood maximization [5][6][7][8]. Once the parameters are trained each hidden unit becomes selectively activated by a specific data feature; owe to the bidirectionality of connections the probability to generate configurations of the visible layer where this feature is present is, in turn, increased. Multiple combinations of numbers of features, with varying degrees of activation of the corresponding hidden units allow for efficient generation of a large variety of new data samples. However, the existence of such 'compositional' encoding seems to depend on the values of the RBM parameters, such as the size of the hidden layer [9]. Characterizing the conditions under which RBM can operate in this compositional regime is the purpose of the present work.In the RBM shown in Fig. 1(a) the visible layer includes N units v i , with i = 1, . . . , N , chosen here to be binary (= 0, 1). Visible units are connected to M hidden units h µ , through the weights {w iµ }. The energy of a configuration v = {v i }, h = {h µ } is defined throughwhere U µ is a potential acting on hidden unit µ; due 2 ; U Ber (h) = h θB if h = 0 or 1, and +∞ otherwise; U ReLU (h) = h 2 2 + h θ for h ≥ 0, +∞ for h < 0. (c) The three regimes of operation, see text. Black, grey and white hidden units symbolize, respectively, strong, weak and null activations....

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Emergence of Compositional Representations in Restricted Boltzmann Machines

2017

View full text Add to dashboard Cite

show abstract

“…We hope that predictions from our model investigation can inspire testable hypotheses for electrophysiological experiments in the future. Further work on this topic includes investigating the neuronal response to multiple suparthreshold periodic signals, and investigating possibles roles of frequency-difference-dependent SR in regulating complicated neurodynamics [19,54,55].…”

Section: Discussionmentioning

confidence: 99%

Frequency-difference-dependent stochastic resonance in neural systems

et al. 2017

View full text Add to dashboard Cite

Biological neurons receive multiple noisy oscillatory signals, and their dynamical response to the superposition of these signals is of fundamental importance for information processing in the brain. Here we study the response of neural systems to the weak envelope modulation signal, which is superimposed by two periodic signals with different frequencies. We show that stochastic resonance occurs at the beat frequency in neural systems at the single-neuron as well as the population level. The performance of this frequency-difference-dependent stochastic resonance is influenced by both the beat frequency and the two forcing frequencies. Compared to a single neuron, a population of neurons is more efficient in detecting the information carried by the weak envelope modulation signal at the beat frequency. Furthermore, an appropriate fine-tuning of the excitationinhibition balance can further optimize the response of a neural ensemble to the superimposed signal. Our results thus introduce and provide insights into the generation and modulation mechanism of the frequencydifference-dependent stochastic resonance in neural systems.

show abstract

Ferromagnetic Models for Cooperative Behavior: Revisiting Universality in Complex Phenomena

Agliari

Barra

Galluzzi

et al. 2014

Mathematical Models and Methods for Planet Earth

Self Cite

View full text Add to dashboard Cite

Ferromagnetic models are harmonic oscillators in statistical mechanics. Beyond their original scope in tackling phase transition and symmetry breaking in theoretical physics, they are nowadays experiencing a renewal applicative interest as they capture the main features of disparate complex phenomena, whose quantitative investigation in the past were forbidden due to data lacking. After a streamlined introduction to these models, suitably embedded on random graphs, aim of the present paper is to show their importance in a plethora of widespread research fields, so to highlight the unifying framework reached by using statistical mechanics as a tool for their investigation. Specifically we will deal with examples stemmed from sociology, chemistry, cybernetics (electronics) and biology (immunology).

show abstract

Multitasking Associative Networks

Cited by 110 publications

References 17 publications

Emergence of Compositional Representations in Restricted Boltzmann Machines

Emergence of Compositional Representations in Restricted Boltzmann Machines

Frequency-difference-dependent stochastic resonance in neural systems

Ferromagnetic Models for Cooperative Behavior: Revisiting Universality in Complex Phenomena

Contact Info

Product

Resources

About