Improving Fairness for Data Valuation in Horizontal Federated Learning

Abstract

Federated learning is an emerging decentralized machine learning scheme that allows multiple data owners to work collaboratively while ensuring data privacy. The success of federated learning depends largely on the participation of data owners. To sustain and encourage data owners' participation, it is crucial to fairly evaluate the quality of the data provided by the data owners and reward them correspondingly. Federated Shapley value, recently proposed by Wang et al. [Federated Learning, 2020], is a measure for data value under the framework of federated learning that satisfies many desired properties for data valuation. However, there are still factors of potential unfairness in the design of federated Shapley value because two data owners with the same local data may not receive the same evaluation. We propose a new measure called completed federated Shapley value to improve the fairness of federated Shapley value. The design depends on completing a matrix consisting of all the possible contributions by different subsets of the data owners. It is shown under mild conditions that this matrix is approximately low-rank by leveraging concepts and tools from optimization. Both theoretical analysis and empirical evaluation verify that the proposed measure does improve fairness in many circumstances.

Fair and efficient contribution valuation for vertical federated learning

“…Federated Shapley value does not need model retraining and preserves some but not all of the favorable qualities of the traditional Shapley value. Fan et al [9] further improved the fairness of this approach.…”

“…Although the Shapley value has many desirable characteristics, its evaluation in the FL context requires repeatedly training and evaluating a machine learning model on all possible subsets of clients. The corresponding communication and computational costs are exponential, and thus prohibitive in practice [29,34,9].…”

“…Computation of the FedSV does not need model retraining and retains some, but not all, of Shapley's fairness criteria. Fan et al [9] further improved the fairness of this approach by leveraging low-rank matrix factorization techniques.…”

Incentivizing Data Contribution in Cross-Silo Federated Learning

“…To achieve this, the central server needs to effectively and fairly evaluate the clients' contribution. Existing studies mainly focus on three types of contribution evaluation/profit allocation mechanisms: linear proportional [11], leave-one-out [10], [12], and Shapley-value-based methods [13], [20], [21]. However, these papers focus on deriving efficient and fair algorithms for contribution valuation/profit allocation, rather than analyzing their impact on the clients' free rider behaviors.…”

“…A reasonable profit allocation mechanism should reward those who contribute more data in training to incentivize data contribution and eliminate free riders. In this paper, we consider three widely adopted profit allocation mechanisms, i.e., linearly proportional (LP) where each client's contribution is measured proportionally to its data size [11]; leave-one-out (LOO) where the contribution is calculated by the difference of the global model accuracy between with and without the client's participation [12]; and Shapley value (SV) where the contribution is measured by the average marginal influence on the global model accuracy [13].…”