An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

Sim, Khe Chai; Zadrazil, Petr; Beaufays, Françoise

doi:10.48550/arxiv.1909.06678

Cited by 8 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For our method to work, we need to assume that S < min{d, C}, i.e., the method works when ∆W is aggregated from the weight updates computed from a reasonable number of samples. This is justifiable, since when running distributed training on private data on edge devices, it is a good practice to use a small batch size [17]. For sequence models that output natural language, d and C are usually in the order of thousands or tens of thousands, which is often much larger than S.…”

Section: Proposed Methodsmentioning

confidence: 99%

Revealing and Protecting Labels in Distributed Training

Dang

Thakkar

Ramaswamy

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Distributed learning paradigms such as federated learning often involve transmission of model updates, or gradients, over a network, thereby avoiding transmission of private data. However, it is possible for sensitive information about the training data to be revealed from such gradients. Prior works have demonstrated that labels can be revealed analytically from the last layer of certain models (e.g., ResNet), or they can be reconstructed jointly with model inputs by using Gradients Matching [1] with additional knowledge about the current state of the model. In this work, we propose a method to discover the set of labels of training samples from only the gradient of the last layer and the id to label mapping. Our method is applicable to a wide variety of model architectures across multiple domains. We demonstrate the effectiveness of our method for model training in two domains -image classification, and automatic speech recognition. Furthermore, we show that existing reconstruction techniques improve their efficacy when used in conjunction with our method. Conversely, we demonstrate that gradient quantization and sparsification can significantly reduce the success of the attack.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

Revealing and Protecting Labels in Distributed Training

Dang

Thakkar

Ramaswamy

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Sim et al [29] found personalizing a model with a user's contact list improves named entity recall from 2.4% to 73.5% -a massive improvement. Personalizing models to individual users with speech disorders improves word error rates by 64% relative [30]. Personalization can make a huge difference in the quality of the recognition, particularly for groups or domains that are underrepresented in the training data.…”

Section: Personalizationmentioning

confidence: 99%

The History of Speech Recognition to the Year 2030

Hannun

2021

Preprint

View full text Add to dashboard Cite

The decade from 2010 to 2020 saw remarkable improvements in automatic speech recognition. Many people now use speech recognition on a daily basis, for example to perform voice search queries, send text messages, and interact with voice assistants like Amazon Alexa and Siri by Apple. Before 2010 most people rarely used speech recognition. Given the remarkable changes in the state of speech recognition over the previous decade, what can we expect over the coming decade? I attempt to forecast the state of speech recognition research and applications by the year 2030. While the changes to general speech recognition accuracy will not be as dramatic as in the previous decade, I suggest we have an exciting decade of progress in speech technology ahead of us.

show abstract

“…Federated learning (FL) enables deep learning models to learn from decentralized data without compromising privacy [28,40]. Various research directions have been explored to investigate this promising field, such as security analysis for FL [5,6,41], efficient communication for FL [3,25,30], personalization for FL [26,39,52], etc.…”

Section: Related Workmentioning

confidence: 99%

PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization

Liu

Guo

Chen

2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) has become a prevalent distributed machine learning paradigm with improved privacy. After learning, the resulting federated model should be further personalized to each different client. While several methods have been proposed to achieve personalization, they are typically limited to a single local device, which may incur bias or overfitting since data in a single device is extremely limited. In this paper, we attempt to realize personalization beyond a single client. The motivation is that during the FL process, there may exist many clients with similar data distribution, and thus the personalization performance could be significantly boosted if these similar clients can cooperate with each other. Inspired by this, this paper introduces a new concept called federated adaptation, targeting at adapting the trained model in a federated manner to achieve better personalization results. However, the key challenge for federated adaptation is that we could not outsource any raw data from the client during adaptation, due to privacy concerns. In this paper, we propose PFA, a framework to accomplish Privacy-preserving Federated Adaptation. PFA leverages the sparsity property of neural networks to generate privacy-preserving representations and uses them to efficiently identify clients with similar data distributions. Based on the grouping results, PFA conducts an FL process in a group-wise way on the federated model to accomplish the adaptation. For evaluation, we manually construct several practical FL datasets based on public datasets in order to simulate both the class-imbalance and background-difference conditions. Extensive experiments on these datasets and popular model architectures demonstrate the effectiveness of PFA, outperforming other state-of-the-art methods by a large margin while ensuring user privacy. We will release our code at: https:// github.com/ lebyni/ PFA.

show abstract

An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

Cited by 8 publications

References 0 publications

Revealing and Protecting Labels in Distributed Training

Revealing and Protecting Labels in Distributed Training

The History of Speech Recognition to the Year 2030

PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization

Contact Info

Product

Resources

About