FLightNER: A Federated Learning Approach to Lightweight Named-Entity Recognition

Abstract

We introduce FLightNER, a collaboratively-trained model in Federated Learning (FL) setting that extends an existing state-of-the-art Named-Entity Recognition (NER) model that uses prompt-tuning known as LightNER. To the best of our knowledge at the time of writing, this is the first work that adapts and evaluates prompt-tuning in an FL environment for NER. FLightNER allows the aggregation of only the trainable parameters of LightNER without model accuracy degradation and saves 10 GB per client by not aggregating the entire model parameters. Trainable-only aggregation enables more clients to join a federation without extending memory prerequisites of the central server.We evaluate our approach against two baselines using three diverse datasets with different distributions across up to seven clients in a federation. We empirically show that compared to the centrally-trained LightNER model, FLightNER outperforms it by 19% when evaluated on a medical dataset, VAERS, with label imbalance across clients. We also show that FLightNER matches the same performance of a centrallytrained LightNER on two balanced datasets: CoNLL and I2B2. Furthermore, we implement and evaluate a federated strategy that drops the outlier client at the farthest distance from the federation's average. We show that dropping the outlier client consistently enhances the performance for the majority of clients.