FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learning

Abstract

Large language models (LLMs) have demonstrated great capabilities in various natural language understanding and generation tasks. Platforms such as Hugging Face facilitate access and utilization of the pre-trained LLMs for different entities, ranging from computer science researchers to users with little machine learning background. Different entities can further improve the performance of those LLMs on their specific downstream tasks by fine-tuning LLMs. When several entities have similar interested tasks, but their local data cannot be shared directly because of privacy concerns regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities. Besides avoiding direct data sharing, FL can also achieve rigorous data privacy protection, model intelligent property protection, and model customization via composition with different techniques. However, fine-tuning LLMs in federated learning settings still lacks adequate support from the existing FL frameworks because it has to deal with optimizing the consumption of significant communication and computational resources, various data preparation for different tasks, and distinct information protection demands. This paper first discusses these challenges of federated fine-tuning LLMs in detail, and introduces our implemented package FederatedScope-LLM (FS-LLM) as a main contribution, which consists of the following components: (1) we build a complete end-to-end benchmarking pipeline, automizing the processes of dataset preprocessing, federated fine-tuning execution or simulation, and performance evaluation on federated LLM fine-tuning with different capability demonstration purposes; (2) we provide comprehensive and off-the-shelf federated parameterefficient fine-tuning (PEFT) algorithm implementations and versatile programming interfaces for future extension to enhance the capabilities of LLMs in FL scenarios with low communication and computation costs, even without accessing the full model (e.g., closed-source LLMs); (3) we adopt several accelerating operators and resource-efficient operators for fine-tuning LLMs with limited resources and the flexible pluggable sub-routines for interdisciplinary study (e.g., LLMs in personalized FL). We conduct extensive and reproducible experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-theart parameter-efficient fine-tuning algorithms in a federated setting, which also yields many valuable insights into federated fine-tuning LLMs for the research community. To facilitate further research and adoption, we release FS-LLM at https://github.com/alibaba/FederatedScope/tree/llm. 1

FedGTA: Topology-Aware Averaging for Federated Graph Learning

“…Graph data has been demonstrated to be superior in multiple applications. Recently, FGL [19,32,45] has received a lot of attention due to its unique advantage in training GNN models collaboratively without sharing collected data for safety concerns.…”

Federated Graph Machine Learning: A Survey of Concepts, Techniques, and Applications

“…Although a number of platforms [3; 8; 146] facilitate FL applications in multiple domains such as vision and language, only a few of them provide off-the-shelf supports for graph datasets and graph machine learning models. To the best of our knowledge, FedGraphNN [39] and FederatedScope-GNN (FS-G) [111] are the two platforms supporting tasks in FGML among the existing FL platforms. FedGraphNN.…”

“…Compared with abundant benchmarks and platforms in standard FL [146; 3; 8; 118], applicable benchmarks and platforms in FGML are still at an infant stage. The current two platforms, FederatedScope-GNN [111] and FedGraphNN [39], lack either graph datasets or off-the-shelf FGML algorithms. In addition, splitting mechanisms in FGML are significantly different from those in standard FL, especially when a graph is split into multiple subgraphs across clients.…”

GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Servers

“…The design rationale behind FL generally puts privacy into the highest priority, targeting the scenario wherein the distributed graph data are possessed by different organizations [54]- [56]. Yet there are many works contemplate other dimensions for federated GNNs: accelerating the training convergence by tackling clients' heterogeneity and communication efficiency [57], [58], boosting the training accuracy upon the not Identically and Independently Distributed (non-IID) graph data [59]- [61], and ensuring the security and fairness of cross-silo data aggregation [62], [63], etc. Nonetheless, DGPE significantly diverges from the FL category in that: 1) DGPE considers distributed GNN inference processing services with a variety of cost optimization objectives, allowing attainable data sharing across edge servers, while FL focuses only on the GNN model training and particularly stresses user privacy by physical data isolation.…”