An Investigation of How Neural Networks Learn from the Experiences of Peers Through Periodic Weight Averaging

Abstract

We investigate a method, weighted average model fusion, that enables neural networks to learn from the experiences of other networks, as well as from their own experiences. This method is inspired by the the social natural of humans, which has been shown to be one of the biggest factors in the development of our cognitive abilities. Modern machine learning has focuses predominantly on learning from direct training, and has largely ignored learning through social engagement with peers, neural networks will the same topology. In order to explore learning through engagement with peers, we have created a way for neural networks to teach each other. Our method allows neural networks to exchange knowledge by combining their weights. It calculates a pairwise weighted average of the weights of two neural networks, and then replaces the existing weight with the new value. We find that weighted average model fusion successfully enables neural networks to learn from the experiences of their peers and combine it with the knowledge that is gained from its own individual experiences. Additionally, we explore the effects that several meta-parameters have on model fusion to provide deeper insights into how the behaves in a variety of scenarios.

On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks

“…It was empirically found that vanilla averaging combines the knowledge contained in several neural networks into a single fused model [35,32]. The simple vanilla averaging, however, only works in the case when the weights of individuals networks are relatively close in the weight space.…”

“…The further details of skill transfer are given in Appendix D.1. We change the weight proportion of model B (also known as fusion rate in [32]) to find the weight combination that has the best performance.…”

“…The simplest technique in model fusion is vanilla averaging, which computes the weighted average of pre-trained network parameters without the need for retraining [35,32]. Their methods fuse neural networks with the same architecture without considering the permutation invariance nature of neural networks.…”

Non-iterative Knowledge Fusion in Deep Convolutional Neural Networks

“…For similar set of weights of the convolutional part, we cannot use the summation (3) as both terms in this expression would have similar values. Instead, the approach of (Smith & Gashler, 2017;Utans, 1996) would be ideal in this case and we can use weight averaging. However, the top fully connected parts still can be fused by weights summation.…”

“…As discussed in Section 2.4, for WS method we used slightly different procedure as before. As the weights of the convolutional parts of both networks originated from the same set of pretrained weights, during fusion by WS method, those weights were averaged according to approach adopted by Smith & Gashler (2017), Utans (1996). The weights of the classifiers on top of networks were obtained from an independent set of weights and thus were summed up as in all previous examples.…”