Fundamental aspects of noise in analog-hardware neural networks

Abstract

We study and analyze the fundamental aspects of noise propagation in recurrent as well as deep, multi-layer networks. The main focus of our study are neural networks in analogue hardware, yet the methodology provides insight for networks in general. The system under study consists of noisy linear nodes, and we investigate the signal-to-noise ratio at the network's outputs which is the upper limit to such a system's computing accuracy. We consider additive and multiplicative noise which can be purely local as well as correlated across populations of neurons. This covers the chief internal-perturbations of hardware networks and noise amplitudes were obtained from a physically implemented recurrent neural network and therefore correspond to a real-world system. Analytic solutions agree exceptionally well with numerical data, enabling clear identification of the most critical components and aspects for noise management. Focusing on linear nodes isolates the impact of network connections and allows us to derive strategies for mitigating noise. Our work is the starting point in addressing this aspect of analogue neural networks, and our results identify notoriously sensitive points while simultaneously highlighting the robustness of such computational systems.

Boolean learning under noise-perturbations in hardware neural networks

“…However, in analogue neural hardware, reproducibility as well as robustness to noise and parameter drifts also play an essential role. We start by collecting statistical information and measure 20 (14) curves for the greedy (Markovian) exploration. All measurements started at an identical position W DMD (1) and we therefore focus on the algorithm's exploration of the error-landscape.…”

“…Internal W DOE and readout W DMD (k) connections are, however, realized in passive and fully parallel photonic hardware. As the network is constructed of physical neurons it harbours noise, which can either be additive or multiplicative, as well as correlated or uncorrelated [14]. The main sources of noise in our experiment are the SLM and the camera, in relation to which the illumination laser and output detector can be considered as noiseless, and so are the internal coupling and readout matrices implemented by the DOE and DMD, respectively.…”

“…All relevant noise sources are therefore reservoir-internal, and our following discussion is by no means limited to systems where the readout layer is implement physically. More details about the theoretical treatment and propagation of noise in NNs as well as the individual noise sources and their respective amplitudes and statistics can be found in [14].…”

“…Noise is an inseparable companion of analogue hardware [14], yet the fundamental aspects of optimizing a noisy NN [15][16][17] have so far hardly been exploredneither in experiments [18,19] nor in theory [14]. Here, we investigate the interactions between noise, learning rules and the topology of an error landscape for the first time.…”

“…Randomly changing one readout weight therefore results in Δy out (k) according to a normalized Gaussian distribution with a width of Δy out, learn (k) σ l (k). We have carefully characterized the noise of all elements in our opto-electronic NN, and their accumulated impact upon y out (k) is excellently approximated by Gaussian white noise with a width of Δy out, noise (k) σ n (k) [14]. Readout weights W DMD remain approximately evenly distributed between zeros and ones for all k. Learning does only modify readout connections and therefore neither modifiesx i nor the system's noise, making both independent of learning.…”