SPIDER: Searching Personalized Neural Architecture for Federated Learning

Abstract

Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server due to privacy and regulatory restrictions. Recent advancements in FL use predefined architecture-based learning for all the clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, in this work, we introduce SPIDER, an algorithmic framework that aims to Search PersonalIzed neural architecture for feDERated learning. SPIDER is designed based on two unique features:(1) alternately optimizing one architecture-homogeneous global model (Supernet) in a generic FL manner and one architecture-heterogeneous local model that is connected to the global model by weight sharing-based regularization (2) achieving architecture-heterogeneous local model by a novel neural architecture search (NAS) method that can select optimal subnet progressively using operation-level perturbation on the accuracy value as the criterion. Experimental results demonstrate that SPIDER outperforms other state-of-the-art personalization methods, and the searched personalized architectures are more inference efficient.

PerFedRLNAS: One-for-All Personalized Federated Neural Architecture Search

“…These methods still search for a global model and failed to do personalization. SPIDER (Mushtaq et al 2021) added a customized architecture search phase during the client local computing while the overhead is very large for transmitting the whole supernet and conducting NAS on the local clients. FedRL-NAS (Yao et al 2021) sent only the subspace to each client, but it still only searched for a global model and had three phases: warm-up, searching and training which is also inefficient.…”

“…Federated Neural Architecture Search (FedNAS) (He, Annavaram, and Avestimehr 2020) was proposed to directly conduct neural architecture search (NAS) in the federated learning setting, with which each client can obtain a model of a different structure along with different weights. However, previous work either search for only a global neural architecture shared among all or manually defined clusters of the clients instead of personalized architecture for each client (Yao et al 2021;Garg, Saha, and Dutta 2021;Laskaridis, Fernandez-Marques, and Dudziak 2022) or incrementally carry out a neural architecture search on each client (He, Annavaram, and Avestimehr 2020;Mushtaq et al 2021). Such simple replacements of fixed models in conventional personalized federated learning with an NAS supernet still cannot provide an insight on personalizing models without manual settings.…”

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

“…In each training round, a device trains a different slice of the NN. Federated NAS [35,74,102] techniques are proposed, using subsets of shared common structure to allow for device personalizing, aiming for better accuracy in non-iid cases and efficient models for inference. An advantage compared to previous techniques is that the devices' NN models can be independently optimized for their hardware, however, this comes at the cost of exploring the architecture search space, which can be resource-hungry.…”

Supernet Training for Federated Image Classification under System Heterogeneity

“…Furthermore, such optimization is simple so that the accuracy gap among sub-models should be bridged through some advanced training techniques. Newly, Mushtaq et al [47] apply continuous differentiable relaxation and gradient descent, but it is significantly sensitive to hyperparameter choices.…”