Differentially-private Federated Neural Architecture Search

Abstract

Neural architecture search, which aims to automatically search for architectures (e.g., convolution, max pooling) of neural networks that maximize validation performance, has achieved remarkable progress recently. In many application scenarios, several parties would like to collaboratively search for a shared neural architecture by leveraging data from all parties. However, due to privacy concerns, no party wants its data to be seen by other parties. To address this problem, we propose federated neural architecture search (FNAS), where different parties collectively search for a differentiable architecture by exchanging gradients of architecture variables without exposing their data to other parties. To further preserve privacy, we study differentially-private FNAS (DP-FNAS), which adds random noise to the gradients of architecture variables. We provide theoretical guarantees of DP-FNAS in achieving differential privacy. Experiments show that DP-FNAS can search highly-performant neural architectures while protecting the privacy of individual parties. The code is available at https://github.com/UCSD-AI4H/DP-FNAS

FedProto: Federated Prototype Learning across Heterogeneous Clients

“…In particular, with the assumption of adding a shared toy dataset in the federated setting, these KD-based FL methods can distill knowledge from a teacher model to student models with different model architectures. Some recent studies have also attempted to combine the neural architecture search with federated learning (Zhu, Zhang, and Jin 2020;He, Annavaram, and Avestimehr 2020;Singh et al 2020), which can be applied to discover a customized model architecture for each group of clients with different hardware capabilities and configurations. A collective learning platform is proposed to handle heterogeneous architectures without access to the local training data and architectures in (Hoang et al 2019).…”

Radiological Analysis of COVID-19 Using Computational Intelligence: A Broad Gauge Study

“… Data Privacy : a detailed discussion of data privacy is given in [ 52 ]. To extend on the avenues mentioned there, the use of differentially private federated neural architecture search [ 142 ] is recommended to preserve data privacy. Through this method, a model can be tested on several subsets of data, which contain varied distributions and distinctions from other datasets and in parallel keep any information about the various data samples completely privatized.…”

A Class-Imbalanced Heterogeneous Federated Learning Model for Detecting Icing on Wind Turbine Blades

“…To overcome the model heterogeneity, KD-based are gained more attention by distilling knowledge from the server model to client models, which have different model structures [29], [30]. Some researchers are integrating the network structure search and FL to discover customized models for clients with different computational capabilities [31], [32]. However, most of the methods above focused on a single heterogeneous challenging scenario of FL, and did not fully consider the available computing capability of each client.…”