Friendly Training: Neural Networks Can Adapt Data To Make Learning Easier

Abstract

In the last decade, motivated by the success of Deep Learning, the scientific community proposed several approaches to make the learning procedure of Neural Networks more effective. When focussing on the way in which the training data are provided to the learning machine, we can distinguish between the classic random selection of stochastic gradient-based optimization and more involved techniques that devise curricula to organize data, and progressively increase the complexity of the training set. In this paper, we propose a novel training procedure named Friendly Training that, differently from the aforementioned approaches, involves altering the training examples in order to help the model to better fulfil its learning criterion. The model is allowed to "simplify" those examples that are too hard to be classified at a certain stage of the training procedure. The data transformation is controlled by a developmental plan that progressively reduces its impact during training, until it completely vanishes. In a sense, this is the opposite of what is commonly done in order to increase robustness against adversarial examples, i.e., Adversarial Training. Experiments on multiple datasets are provided, showing that Friendly Training yields improvements with respect to informed data sub-selection routines and random selection, especially in deep convolutional architectures. Results suggest that adapting the input data is a feasible way to stabilize learning and improve the generalization skills of the network.

Being Friends Instead of Adversaries: Deep Networks Learn from Data Simplified by Other Networks

“…Datasets (≈ 60k samples) are already divided into training, validation and test set . We compared the test error rates of the FC-A/B and CNN-A/B models in CT, FT/NFT, and also using the CL-inspired data sorting policy of (Marullo et al 2021), named Easy-Examples First (EEF) that has the same temporal dynamics of FT. Experiments are executed for γ max = 200 epochs, and we selected the model with the lowest validation error considering η max ∈ {500, 1000, 2000}, γ max simp ∈ {0.25, 0.5, 0.85} • γ max , β ∈ {10 −5 , 10 −4 , 5 • 10 −4 }.…”

“…Table 1 reports the test error rate of the different models, where other baseline results exploiting different types of classifier can be found in (Marullo et al 2021) (typically overcame by FT/NFT). Our analysis starts by confirming that the family of Friendly Training algorithms (being them neural or not) very frequently shows better results than CT and of EEF.…”

“…mn-back mn-rot-back mn-rot rectangles convex FC-A CT 28.34±0.09 64.06±0.31 43.16±0.51 24.31±0.21 33.91±0.44 EEF 28.18±0.47 64.27±0.19 43.91±0.73 24.48±0.11 33.17±0.93 FT 28.66±0.06 64.14±0.36 43.24±0.43 24.64±0.37 34.38±0.22 NFT 28.15±0.04 64.55±0.14 42.96±0.58 24.57±0.19 34.25±1.03 Comparison of different classifiers (FC-A, FC-B, CNN-A, CNN-B) and learning algorithms (CT, EEF, FT from(Marullo et al 2021) and our NFT) -datasets of Section 3.1 (where MN stands for MNIST and removing the suffix IMAGE). Test error and standard deviation over 3 runs are reported.…”

Being Friends Instead of Adversaries: Deep Networks Learn from Data Simplified by Other Networks

“…Friendly Training (FT) (Marullo et al 2021) is a novel approach belonging to the latter category. FT allows the training procedure not only to adapt the weights and biases of the classifier, but also to transform the training data in order to facilitate the early fulfilment of the learning criterion.…”

“…These are pretty common situations of every data collection process of the real-world. In the case of CL, this has been recently discussed and evaluated in (Wu, Dyer, and Neyshabur 2020), while in the case of FT the existing evaluation is limited to artificial datasets for digit recognition (Marullo et al 2021).…”