Federated Learning of Artificial Neural Networks

Abstract

The training of the most expressive state-of-the-art Machine Learning (ML) models, and especially that one of Artificial Neural Networks (NN) requires a huge amount of training data, along with very significant computation resources. The paradigm of Federated Learning (FL) focuses on the possibilities of collaborative training of ML models in heterogeneous, spatially distributed environment of recent IT infrastructures, dividing the burden of computation among all the stakeholders and leaving the potentially sensitive data at the location of its creation.In this dissertation I am presenting my work relating to the possibilities to alleviate apparent problems of federated training of NNs. For the experienced performance loss we propose to adapt and use stateful optimization techniques for the FL setup. For communication complications of the centralized training we tested a technique to simulate FL in a peer to peer environment. And finally for the privacy issues we present a method to train NNs in an FL environment via a derivative free genetic algorithm. ContributionChapter 1 ContributionIn the research field of federated optimization, in our view, recent works are conducted with the goal in mind to alleviate some of the three main problems of FL algorithms from the original proposals in [129] and [157]. The first of these problems is the observed performance degradation of the learning process, or insufficient accuracy on the side of the end users. The second research direction is about practical questions of implementing these methods, such that resource provisioning, or more concretely, addressing the communication problems in real world network architectures. And the third group of works is seeking to provide stronger privacy guaranties for the users who participate with their data in the training process.In my theses I propose simple techniques to alleviate the three above mentioned issues of FL. Performance degradationThe first problem we examined is the empirical fact that FL of NNs under-performs central training, both in the accuracy of final models, and in the rate of learning. According to our intuition this issue might be relieved at some extent by stateful optimization methods, that have been originally designed to overcome similar problems to those that arises from the characteristics of FL training. Based on our experiments [64,126], presented in Chapter 4, these methods can help to alleviate the performance issues of FL in almost all the examined cases.