Retrospective Loss: Looking Back to Improve Training of Deep Neural Networks

Jandial, Surgan; Chopra, Ayush; Sarkar, Mausoom; Gupta, Piyush; Krishnamurthy, Balaji; Balasubramanian, Vineeth N

doi:10.1145/3394486.3403165

Cited by 3 publications

(1 citation statement)

References 19 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods can work with existing architectures to improve performance. From another insight, retrospective loss (RL) [18] proposes to utilize the past model state during the training process Take the red boxes as an example…”

Section: Related Workmentioning

confidence: 99%

Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values

Lin

Guo

Xing

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

To obtain good performance, convolutional neural networks are usually over-parameterized. This phenomenon has stimulated two interesting topics: pruning the unimportant weights for compression and reactivating the unimportant weights to make full use of network capability. However, current weight reactivation methods usually reactivate the entire filters, which may not be precise enough. Looking back in history, the prosperity of filter pruning is mainly due to its friendliness to hardware implementation, but pruning at a finer structure level, i.e., weight elements, usually leads to better network performance. We study the problem of weight element reactivation in this paper. Motivated by evolution, we select the unimportant filters and update their unimportant elements by combining them with the important elements of important filters, just like gene crossover to produce better offspring, and the proposed method is called weight evolution (WE). WE is mainly composed of four strategies. We propose a global selection strategy and a local selection strategy and combine them to locate the unimportant filters. A forward matching strategy is proposed to find the matched important filters and a crossover strategy is proposed to utilize the important elements of the important filters for updating unimportant filters. WE is plug-in to existing network architectures. Comprehensive experiments show that WE outperforms the other reactivation methods and plug-in training methods with typical convolutional neural networks, especially lightweight networks. Our code is available at https://github.com/BZQLin/Weight-evolution. CCS CONCEPTS• Computing methodologies → Neural networks.

show abstract

Section: Related Workmentioning

confidence: 99%