Speculative Backpropagation for CNN Parallel Training

Abstract

The parallel learning in neural networks can greatly shorten the training time. Its prior efforts were mostly limited to distributing inputs to multiple computing engines. It is because the gradient descent algorithm in the neural network training is inherently sequential. This paper proposes a novel CNN parallel training method for image recognition. It overcomes the sequential property of the gradient descent and enables the parallel training with the speculative backpropagation. We found that the Softmax and ReLU outcomes in the forward propagation for the same labels are likely to be very similar. This characteristic makes it possible to perform the forward and backward propagation simultaneously. We implemented the proposed parallel model with CNNs in both software and hardware, and evaluated its performance. The parallel training reduces the training time by 34% in CIFAR-100 without the loss of the prediction accuracy compared to the sequential training. In many cases, it even improves the accuracy.

Expert system gradient descent style training: Development of a defensible artificial intelligence technique

“…Techniques which use speculative approaches [49], spiking neural network concepts [50,51] and memory use optimization [52] have also been proposed. Kim and Ko [53] and (separately) Ma, Lewis and Kleijn [54] have proposed techniques which train neural networks while specifically avoiding backpropagation altogether.…”

Machine Learning Methods for Spaceborne GNSS-R Sea Surface Height Measurement From TDS-1

“…The specific process of using CNN to train the SSH inversion model can be described as follows: (1) First, the training data are input into the network in batches for forward propagation; (2) Then, the weights of network are modified by using back propagation learning [28]; (3) The iterative training is carried out continuously and stops after 10000.…”

Automating the design and development of gradient descent trained expert system networks

“…As a relatively mature technique, a variety of approaches have been developed, including those that have been shown to enhance performance speed [77], techniques that attempt to increase accuracy by introducing noise [78] and some that seek to create attack resilience [79,80]. Other techniques have incorporated federated learning and momentum [81] and used evolutionary algorithms [82], speculative approaches [83] and spiking neural network concepts [84,85]. Yet other techniques have focused on supporting deep networks [86], memory use optimization [87], bias factors [88,89] and initial condition sensitivity [90].…”