FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction

Sui, Dianbo; Chen, Yubo; Zhao, Jun; Jia, Yunyi; Xie, Yuantao; Sun, Weijian

doi:10.18653/v1/2020.emnlp-main.165

Cited by 77 publications

(49 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are a few prior works starting to explore federated learning methods in privacy-preserving NLP applications, such as keyboard prediction (Hard et al, 2018;Leroy et al, 2019), intent classification (Zhu et al, 2020), pretraining and fine-tuning language model (Liu and Miller, 2020) and medical name entity recognition (Ge et al, 2020). Sui et al (2020) is most relevant to our work, which applies federated learning to supervised relation classification. But in their work, the data stored by the local platforms must be manually labeled in advance, which is difficult to be satisfied in practical application.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Distantly Supervised Relation Extraction in Federated Settings

Sui

Chen

Liu

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

In relation extraction, distant supervision is widely used to automatically label a largescale training dataset by aligning a knowledge base with unstructured text. Most existing studies in this field have assumed there is a great deal of centralized unstructured text. However, in practice, texts are usually distributed on different platforms and cannot be centralized due to privacy restrictions. Therefore, it is worthwhile to investigate distant supervision in the federated learning paradigm, which decouples the training of the model from the need for direct access to raw texts. However, overcoming label noise of distant supervision becomes more difficult in federated settings, because texts containing the same entity pair scatter around different platforms. In this paper, we propose a federated denoising framework to suppress label noise in federated settings. The key of this framework is a multiple instance learning based denoising method that is able to select reliable sentences via cross-platform collaboration. Various experiments on New York Times dataset and miRNA gene regulation relation dataset demonstrate the effectiveness of the proposed method. 1

show abstract

Section: Related Workmentioning

confidence: 99%

“…But in their work, the data stored by the local platforms must be manually labeled in advance, which is difficult to be satisfied in practical application. Compared with Sui et al (2020), we combine federated learning with distant supervision, which can avoid such a unpractical assumption.…”

Section: Related Workmentioning

confidence: 99%

Distantly Supervised Relation Extraction in Federated Settings

Sui

Chen

Liu

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…To test the proposed approach, we follow the convention of recent FL-based NLP studies (Liu et al, 2019;Huang et al, 2020b;Zhu et al, 2020;Sui et al, 2020) our simulation are heterogeneous), which is similar to the simulation setting of aforementioned previous studies. We split each genre into train/dev/test splits following Wang et al (2011) and report the statistics (in terms of the number of sentences, word tokens, and OOV rate) in Table 2.…”

Section: Simulationsmentioning

confidence: 99%

“…Unfortunately, limited attentions have been paid to address this issue. Most existing approaches (Liu et al, 2019;Huang et al, 2020b;Sui et al, 2020) with FL on NLP (e.g., for language modeling Figure 1: The server-node architecture of our approach. The encrypted information (i.e., encrypted data, word segmentation tags, and loss) communicates between a node and the server, where the locally stored data is inaccessible to other nodes during the training process.…”

Section: Introductionmentioning

confidence: 99%

Federated Chinese Word Segmentation with Global Character Associations

Tian¹,

Chen²,

Qin³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Chinese word segmentation (CWS) is a fundamental task for Chinese information processing, which always suffers from out-ofvocabulary word issues, especially when it is tested on data from different sources. Although one possible solution is to use more training data, in real applications, these data are stored at different locations and thus are invisible and isolated among each other owing to the privacy or legal issues (e.g., clinical reports from different hospitals). To address this issue and benefit from extra data, we propose a neural model for CWS with federated learning (FL) adopted to help CWS deal with data isolation, where a mechanism of global character associations is proposed to enhance FL to learn from different data sources. Experimental results on a simulated environment with five nodes confirm the effectiveness of our approach, where our approach outperforms different baselines including some well-designed FL frameworks. 1

show abstract

“…data settings, i.e., some of the clients have signiicantly diferent data distribution than the others. Other approaches require some trusted clients or samples to guide the learning [27,35,43] or detect the updates from corrupted clients [16,21]. Unfortunately, the credibility of these trusted clients and samples are usually not guaranteed.…”

Section: Introductionmentioning

confidence: 99%

Auto-weighted Robust Federated Learning with Corrupted Data Sources

Ngai

et al. 2022

ACM Trans. Intell. Syst. Technol.

View full text Add to dashboard Cite

Federated learning provides a communication-efficient and privacy-preserving training process by enabling learning statistical models with massive participants without accessing their local data. Standard federated learning techniques that naively minimize an average loss function are vulnerable to data corruptions from outliers, systematic mislabeling, or even adversaries. In this paper, we address this challenge by proposing Auto-weighted Robust Federated Learning ( ARFL ), a novel approach that jointly learns the global model and the weights of local updates to provide robustness against corrupted data sources. We prove a learning bound on the expected loss with respect to the predictor and the weights of clients, which guides the definition of the objective for robust federated learning. We present an objective that minimizes the weighted sum of empirical risk of clients with a regularization term, where the weights can be allocated by comparing the empirical risk of each client with the average empirical risk of the best p clients. This method can downweight the clients with significantly higher losses, thereby lowering their contributions to the global model. We show that this approach achieves robustness when the data of corrupted clients is distributed differently from the benign ones. To optimize the objective function, we propose a communication-efficient algorithm based on the blockwise minimization paradigm. We conduct extensive experiments on multiple benchmark datasets, including CIFAR-10, FEMNIST, and Shakespeare, considering different neural network models. The results show that our solution is robust against different scenarios including label shuffling, label flipping, and noisy features, and outperforms the state-of-the-art methods in most scenarios.

show abstract

FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction

Cited by 77 publications

References 37 publications

Distantly Supervised Relation Extraction in Federated Settings

Distantly Supervised Relation Extraction in Federated Settings

Federated Chinese Word Segmentation with Global Character Associations

Auto-weighted Robust Federated Learning with Corrupted Data Sources

Contact Info

Product

Resources

About