An Investigation into On-Device Personalization of End-to-End Automatic Speech Recognition Models

Abstract

Speaker-independent speech recognition systems trained with data from many users are generally robust against speaker variability and work well for a large population of speakers. However, these systems do not always generalize well for users with very different speech characteristics. This issue can be addressed by building personalized systems that are designed to work well for each specific user. In this paper, we investigate the idea of securely training personalized end-to-end speech recognition models on mobile devices so that user data and models never leave the device and are never stored on a server. We study how the mobile training environment impacts performance by simulating on-device data consumption. We conduct experiments using data collected from speech impaired users for personalization. Our results show that personalization achieved 63.7% relative word error rate reduction when trained in a server environment and 58.1% in a mobile environment. Moving to on-device personalization resulted in 18.7% performance degradation, in exchange for improved scalability and data privacy. To train the model on device, we split the gradient computation into two and achieved 45% memory reduction at the expense of 42% increase in training time.

Federated Learning for Healthcare Informatics

“…Cost and labor savings. Although there are some general work about federated learning personalization [48,94], for healthcare informatics, how to combining the medical domain knowledge and make the global model be personalized for every medical institutions or wearable devices is another open question. -Model Precision.…”

Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

“…Because the size of E2E models is much smaller than that of hybrid models, E2E models have clear advantages when being deployed to device. Therefore, personalization or adaptation of E2E models [119], [120], [126], [127] is a rapidly growing area. While it possible to adapt every user's model on cloud and then push it back to device, it is more reasonable to adapt the model on device, which needs to adjust the adaptation algorithm to overcome the challenge of limited memory and computation power [119].…”

“…Therefore, personalization or adaptation of E2E models [119], [120], [126], [127] is a rapidly growing area. While it possible to adapt every user's model on cloud and then push it back to device, it is more reasonable to adapt the model on device, which needs to adjust the adaptation algorithm to overcome the challenge of limited memory and computation power [119]. Another interesting direction for the adaptation of E2E models is how to leverage unpaired data especially text only data in a new domain.…”

“…Sim et al [119], updated the acoustic encoder of RNN-T models using speaker-specific adaptation data. Furthermore, by generating text-to-speech (TTS) audio from the target speaker, more data can be used to adapt acoustic encoder.…”

“…[117]- [119], [180], [228], [231], [248], [291] Embedding Hybrid [56], [57], [61], [74], [130], [132], [138], [148], [150], [153], [159], [161], [168], [213], [292] E2E [62], [128], [217] Feature Hybrid [56], [74], [75], [135], [138], [168], [230], [285], [289], [290], [293] Data Hybrid [116], [193] embedding classes. The overall RERR is 9.72% 1 .…”