Gradient Learning under Tilted Empirical Risk Minimization

Liu, Liyuan; Song, Biqin; Pan, Zhibin; Yang, Chao; Chi, Xiao; Li, Weifu

doi:10.3390/e24070956

Cited by 9 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…( 4), which behaves similarly to q-FFL. While the two objectives show comparable improvements in the evaluations of Li et al [11], their interactions with personalisation are unknown.…”

Section: Fair Federated Learningmentioning

confidence: 97%

“…A 𝑞 = 0 corresponds to FedAvg, while larger values prioritise higher losses to improve accuracy on clients for which the federated model underperforms. Li et al [11] develop Tilted Empirical Risk Minimization (TERM), shown in Eq. ( 4), which behaves similarly to q-FFL.…”

Section: Fair Federated Learningmentioning

confidence: 99%

“…The original publications of Li et al [11,15] used a weighted sampling of devices based on their number of examples, followed by uniform averaging. However, the server must know the dataset size of all clients a priori, which is potentially unfeasible.…”

Section: Fair Federated Learningmentioning

confidence: 99%

“…The existing body of research on balancing global and local performance has proposed several approaches. Two Fair Federated Learning (FFL) techniques, q-Fair Federated Learning (q-FFL) and Tilted Empirical Risk Minimization (TERM), proposed by Li et al [15] and Li et al [11] respectively, aim to improve the accuracy of the worst-performing clients by prioritising those with large losses during FL. Alternatively, Yu et al [23] and Mansour et al [17] recommend using personalisation (local adaptation) methods such as Freezebase (FB), Elastic Weight Consolidation (EWC), and Knowledge Distillation (KD) for fine-tuning.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Can Fair Federated Learning Reduce the need for Personalisation?

Alex

Gusmão

Lane

2023

Proceedings of the 3rd Workshop on Machine Learning and Systems

View full text Add to dashboard Cite

Federated Learning (FL) enables training ML models on edge clients without sharing data. However, the federated model's performance on local data varies, disincentivising the participation of clients who benefit little from FL. Fair FL reduces accuracy disparity by focusing on clients with higher losses while personalisation locally fine-tunes the model. Personalisation provides a participation incentive when an FL model underperforms relative to one trained locally. For situations where the federated model provides a lower accuracy than a model trained entirely locally by a client, personalisation improves the accuracy of the pre-trained federated weights to be similar to or exceed those of the local client model. This paper evaluates two Fair FL (FFL) algorithms as starting points for personalisation. Our results show that FFL provides no benefit to relative performance in a language task and may double the number of underperforming clients for an image task. Instead, we propose Personalisation-aware Federated Learning (PaFL) as a paradigm that pre-emptively uses personalisation losses during training. Our technique shows a 50% reduction in the number of underperforming clients for the language task while lowering the number of underperforming clients in the image task instead of doubling it. Thus, evidence indicates that it may allow a broader set of devices to benefit from FL and represents a promising avenue for future experimentation and theoretical analysis.

show abstract

“…( 4), which behaves similarly to q-FFL. While the two objectives show comparable improvements in the evaluations of Li et al [11], their interactions with personalisation are unknown.…”

Section: Fair Federated Learningmentioning

confidence: 97%

Section: Fair Federated Learningmentioning

confidence: 99%

Section: Fair Federated Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Can Fair Federated Learning Reduce the need for Personalisation?

Alex

Gusmão

Lane

2023

Proceedings of the 3rd Workshop on Machine Learning and Systems

View full text Add to dashboard Cite

show abstract

“…where w i ą 0 denote the respective weight coefficients; see, also, Li et al (2021a) and Li et al (2021b) for alternative loss functionals inspired by fair resource allocation (Moulin 2003). One of the most popular FL algorithms, FedAvg FL is strikingly similar to a voting system, which is one of the most studied scenarios in social choice theory.…”

Section: Federated Learningmentioning

confidence: 99%

Impossibility of Collective Intelligence

Muandet¹

2022

Preprint

View full text Add to dashboard Cite

Democratization of AI involves training and deploying machine learning models across heterogeneous and potentially massive environments. Diversity of data opens up a number of possibilities to advance AI systems, but also introduces pressing concerns such as privacy, security, and equity that require special attention. This work shows that it is theoretically impossible to design a rational learning algorithm that has the ability to successfully learn across heterogeneous environments, which we decoratively call collective intelligence (CI). By representing learning algorithms as choice correspondences over a hypothesis space, we are able to axiomatize them with essential properties. Unfortunately, the only feasible algorithm compatible with all of the axioms is the standard empirical risk minimization (ERM) which learns arbitrarily from a single environment. Our impossibility result reveals informational incomparability between environments as one of the foremost obstacles for researchers who design novel algorithms that learn from multiple environments, which sheds light on prerequisites for success in critical areas of machine learning such as out-of-distribution generalization, federated learning, algorithmic fairness, and multi-modal learning.

show abstract

Federated learning with superquantile aggregation for heterogeneous data

et al. 2023

View full text Add to dashboard Cite

We present a federated learning framework that is designed to robustly deliver good predictive performance across individual clients with heterogeneous data. The proposed approach hinges upon a superquantile-based learning objective that captures the tail statistics of the error distribution over heterogeneous clients. We present a stochastic training algorithm that interleaves differentially private client filtering with federated averaging steps. We prove finite time convergence guarantees for the algorithm: O(1∕ √ T) in the nonconvex case in T communication rounds and O(exp(−T∕ 3∕2 ) + ∕T) in the strongly convex case with local condition number . Experimental results on benchmark datasets for federated learning demonstrate that our approach is competitive with classical ones in terms of average error and outperforms them in terms of tail statistics of the error.

show abstract

Gradient Learning under Tilted Empirical Risk Minimization

Cited by 9 publications

References 25 publications

Can Fair Federated Learning Reduce the need for Personalisation?

Can Fair Federated Learning Reduce the need for Personalisation?

Impossibility of Collective Intelligence

Federated learning with superquantile aggregation for heterogeneous data

Contact Info

Product

Resources

About