Robustness of hardware-oriented restricted Boltzmann machines in deep belief networks for reliable processing

Abstract

Remarkable hardware robustness of deep learning is revealed from an error-injection analysis performed using a custom hardware model implementing parallelized restricted Boltzmann machines (RBMs). RBMs used in deep belief networks (DBNs) demonstrate robustness against memory errors during and after learning. Fine-tuning has a significant impact on the recovery of accuracy under the presence of static errors that may modify structural data of RBMs. The proposed hardware networks with fine-graded memory distribution are observed to tolerate memory errors, thereby resulting in a reliable deep learning hardware platform, potentially suitable to safety-critical embedded applications.

Artificial neuron operations and spike-timing-dependent plasticity using memristive devices for brain-inspired computing

“…1) The hardware implementation of RBMs has been studied, and scalable and highly parallel RBM microelectronic systems have been developed and analyzed. [4][5][6] In layer-by-layer RBM structures, the stochastic iterative learning efficiently reduces computation time, although all the learned data such as connection weights and node biases must be stored in memory and updated during learning. Figure 2(b) shows that the average number of computation steps is thereby reduced to half the default value in RBM learning with the MNIST digit dataset.…”

“…A drawback of such high-performance systems is that they consume more power than the field-programmable gate array (FPGA) and application-specific integrated circuit implementations of DL. [3][4][5][6] The power area density of the chips limits the total computing acceleration. To satisfy the growing DL needs of data scientists, more efficient hardware systems are required.…”

Integrated analog neurons inspired by mimicking synapses with metal-oxide memristive devices

“…There is much room for additional learning in the same hardware to recover from such data drift. [5][6][7] Before discussing NN inference circuits, we tried to simplify the learning sequence of the NN by focusing on backward connections between the hidden layer and the output layer. 26) A simple mechanism, called "random backpropagation NN," has been presented.…”

“…A drawback of such highperformance systems based on GP computers is that they are more power-hungry than the custom-made logic circuits in a field programmable gate array or application-specific integrated circuit. [4][5][6][7] To implement the intelligence of NNs on tiny systems such as recent Internet of Things (IoT) devices, the computation power must be reduced by using novel hardware architectures and calculation cores. Recently, binary connections have been applied to DNN computing in timedomain processing architecture to enable time-and energyefficient computing in a fully spatially unrolled architecture utilizing two-terminal memristive cells in a small circuit.…”