Parallel learning by multitasking neural networks

Abstract

Parallel learning, namely the simultaneous learning of multiple patterns, constitutes a modern challenge for neural networks. While this cannot be accomplished by standard Hebbian associative neural networks, in this paper we show how the multitasking Hebbian network (a variation on the theme of the Hopfield model, working on sparse datasets) is naturally able to perform this complex task. We focus on systems processing in parallel a finite (up to logarithmic growth in the size of the network) number of patterns, mirroring the low-storage setting of standard associative neural networks. When patterns to be reconstructed are mildly diluted, the network handles them hierarchically, distributing the amplitudes of their signals as power laws w.r.t. the pattern information content (hierarchical regime), while, for strong dilution, the signals pertaining to all the patterns are simultaneously raised with the same strength (parallel regime). Further, we prove that the training protocol (either supervised or unsupervised) neither alters the multitasking performances nor changes the thresholds for learning. We also highlight (analytically and by Monte Carlo simulations) that a standard cost function (i.e. the Hamiltonian) used in statistical mechanics exhibits the same minima as a standard loss function (i.e. the sum of squared errors) used in machine learning.

Replica symmetry breaking in supervised and unsupervised Hebbian networks

“…From the mathematical viewpoint, possible future developments would be relaxing the constraints of a self-averaging Mattis magnetization and inspecting how the learning and retrieval properties of these networks change with different kind of noise: in this work we have focused on multiplicative noise, however the use of additive noise has lately gained large popularity in generative models for machine learning (see for instance [56]). Furthermore, within the framework of multiplicative noise, an interesting outlook would be considering corruptions of archetypes which also consist of blank (in addition to inverted) entries, as done recently in [57] for networks away from saturation. Their operation in the saturated regime and the effects of replica symmetry breaking have not yet been investigated: we plan to report soon on these topics.…”