GTG-Shapley: Efficient and Accurate Participant Contribution Evaluation in Federated Learning

Abstract

Federated Learning (FL) bridges the gap between collaborative machine learning and preserving data privacy. To sustain the long-term operation of an FL ecosystem, it is important to attract high-quality data owners with appropriate incentive schemes. As an important building block of such incentive schemes, it is essential to fairly evaluate participants’ contribution to the performance of the final FL model without exposing their private data. Shapley Value (SV)-based techniques have been widely adopted to provide a fair evaluation of FL participant contributions. However, existing approaches incur significant computation costs, making them difficult to apply in practice. In this paper, we propose the Guided Truncation Gradient Shapley (GTG-Shapley) approach to address this challenge. It reconstructs FL models from gradient updates for SV calculation instead of repeatedly training with different combinations of FL participants. In addition, we design a guided Monte Carlo sampling approach combined with within-round and between-round truncation to further reduce the number of model reconstructions and evaluations required, through extensive experiments under diverse realistic data distribution settings. The results demonstrate that GTG-Shapley can closely approximate actual Shapley values, while significantly increasing computational efficiency compared to the state of the art, especially under non-i.i.d. settings.

Contribution-Aware Federated Learning for Smart Healthcare

“…In this section, we describe the AI Engine of CAreFL. It can be divided into two main parts: 1) an efficient Shapley Value (SV)-based participant contribution evaluation algorithm -GTG-Shapley (Liu et al 2022), and 2) a contribution-aware FL model aggregation algorithm. The system architecture of the AI Engine is illustrated in Figure 3.…”

“…In the FL for smart healthcare application scenario, a contribution evaluation solution that can fairly assess participants' contributions in a highly efficient manner is required. Therefore, the CAreFL AI Engine is incorporated with our proposed Guided Truncation Gradient Shapley (GTG-Shapley) approach (Liu et al 2022). It not only significantly improves computation efficiency, but also achieves higher accuracy compared to the state-of-the-art SV-based FL participant contribution evaluation approaches.…”

“…The Thirty-Sixth AAAI Conference on Artificial Intelligence rated with our proposed GTG-Shapley (Liu et al 2022) approach, which can evaluate fair and accurate FL participant contribution in a highly efficient manner.…”

Towards More Efficient Data Valuation in Healthcare Federated Learning Using Ensembling

“…The few studies exploring computation of SV for cross-device FL settings [24,35,37], cannot consider contributions from all devices at the same time. In particular, this limits the applicability of these methods for cross-silo settings, which is relevant to medical use cases where all institutions contribute at the same time.…”

CAreFL: Enhancing smart healthcare with Contribution‐Aware Federated Learning

“…To address these issues, we present the Contribution-Aware Federated Learning (CAreFL) framework in this paper. Built on top of the GTG-Shapley FL client contribution evaluation (CE) method we have previously developed (Liu et al 2022), it provides fair and accurate FL participant CE efficiently without the need to directly inspect the original data. In addition, it leverages the knowledge about the performance of various sub-models generated during Shapley value (SV) estimation to offer a new "survival of the fittest" FL model aggregation approach.…”