Decoupled Parallel Backpropagation with Convergence Guarantee

Huo, Zhouyuan; Gu, Bin; Yang, Qian; Huang, Heng

doi:10.48550/arxiv.1804.10574

Cited by 12 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 1 We remark that a different definition of the speedup ratio has been applied by (Huo et al, 2018b). That is, by terminating the parallel training process once its testing accuracy is comparable to that of layer-serial training method, the speedup can then be defined as the ratio of serial execution time to the parallel execution time.…”

Section: Methodsmentioning

confidence: 99%

“…We then compare our joint learning approach with other representative methods from literature for solving the image classification task (1) across CIFAR-10 dataset. Recall from Table 2 or Table 1 that the algorithmic locking issues of BP are not addressed by neither PipeDream nor Gpipe while the loss function of DGL methods is not consistent with the original learning task, the comparison is therefore made against the straight-forward implementation of penalty method without using data augmentation (Gotmare et al, 2018), FR (Huo et al, 2018a), DDG (Huo et al, 2018b) and BP (He et al, 2016a). We follow exactly the same setup from (Huo et al, 2018b,a) and report the experimental results in Table 9.…”

Section: Comparison To Other Methodsmentioning

confidence: 99%

“…We follow exactly the same setup from (Huo et al, 2018b,a) and report the experimental results in Table 9. It is noteworthy that in contrast to the definition of speedup ratio applied in (Huo et al, 2018b), we adopt the more commonly used definition (27) for measuring the speed gain of parallel processing. Under such a circumstance, both DDG and FR show no speedup over the traditional layer-serial method (see…”

Section: Comparison To Other Methodsmentioning

confidence: 99%

“…Another kind of method relies on the employment of delayed gradients for updating network parameters and therefore achieves backward unlocking (Huo et al, 2018b). However, this design suffers from large memory consumption and weight staleness introduced by the asynchronous backward updates, especially when the network goes deeper (Lian et al, 2015;.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks

Sun¹,

Dong²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Gradient-based methods for the distributed training of residual networks (ResNets) typically require a forward pass of the input data, followed by back-propagating the error gradient to update model parameters, which becomes time-consuming as the network goes deeper. To break the algorithmic locking and exploit synchronous module parallelism in both the forward and backward modes, auxiliary-variable methods have attracted much interest lately but suffer from significant communication overhead and lack of data augmentation. In this work, a novel joint learning framework for training realistic ResNets across multiple compute devices is established by trading off the storage and recomputation of external auxiliary variables. More specifically, the input data of each independent processor is generated from its low-capacity auxiliary network (AuxNet), which permits the use of data augmentation and realizes forward unlocking. The backward passes are then executed in parallel, each with a local loss function that originates from the penalty or augmented Lagrangian (AL) methods. Finally, the proposed AuxNet is employed to reproduce the updated auxiliary variables through an end-to-end training process. We demonstrate the effectiveness of our methods on ResNets and WideResNets across CIFAR-10, CIFAR-100, and ImageNet datasets, achieving speedup over the traditional layer-serial training method while maintaining comparable testing accuracy.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Comparison To Other Methodsmentioning

confidence: 99%

Section: Comparison To Other Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks

Sun¹,

Dong²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We survey three categories of BP literature-(i) better hardware implementation of BP [15,16,31,11,32,25], (ii) workarounds to approximate BP [33,7,10], and (iii) biologically inspired algorithms. Biologically inspired algorithms can further be segregated into four types: (i) Inspired from biological observations [29,7,26,17], these works try to approximate BP with the intention resolve its biological implausibility, (ii) Propagation of an alternative to error [19,21], (iii) Leveraging local errors, the power of single layer networks, and layer wise pre-training to approximate BP [24,23,3], (iv) Resolving the locking problem using decoupling [14,6,12,1,20] and its variants [27,8,22,4]. We were deeply motivated by (ii), (iii), and (iv) while coming up with the idea of 'front contributions'-specifically, propagating something other than error, the idea of a single layer network, and decoupling, collectively inspire 'front contributions'.…”

Section: Introduction and Related Workmentioning

confidence: 99%

Front Contribution instead of Back Propagation

Mishra¹,

Arunkumar²

2021

Preprint

View full text Add to dashboard Cite

Deep Learning's outstanding track record across several domains has stemmed from the use of error backpropagation (BP). Several studies, however, have shown that it is impossible to execute BP in a real brain. Also, BP still serves as an important and unsolved bottleneck for memory usage and speed. We propose a simple, novel algorithm, the Front-Contribution algorithm, as a compact alternative to BP. The contributions of all weights with respect to the final layer weights are calculated before training commences and all the contributions are appended to weights of the final layer, i.e., the effective final layer weights are a non-linear function of themselves. Our algorithm then essentially collapses the network, precluding the necessity for weight updation of all weights not in the final layer. This reduction in parameters results in lower memory usage and higher training speed. We show that our algorithm produces the exact same output as BP, in contrast to several recently proposed algorithms approximating BP. Our preliminary experiments demonstrate the efficacy of the proposed algorithm. Our work provides a foundation to effectively utilize these presently under-explored "front contributions", and serves to inspire the next generation of training algorithms.

show abstract

Random feedback alignment algorithms to train neural networks: why do they align?

Chu,

Bacho

2024

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. The performance of FA is generally attributed to an alignment of the update of the random walker with the true gradient --- the eponymous gradient alignment --- which drives an approximate gradient descent. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descent.

show abstract

Decoupled Parallel Backpropagation with Convergence Guarantee

Cited by 12 publications

References 0 publications

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks

Front Contribution instead of Back Propagation

Random feedback alignment algorithms to train neural networks: why do they align?

Contact Info

Product

Resources

About