Improved Robustness to Disfluencies in RNN-Transducer Based Speech Recognition

Abstract

Automatic Speech Recognition (ASR) based on Recurrent Neural Network Transducers (RNN-T) is gaining interest in the speech community. We investigate data selection and preparation choices aiming for improved robustness of RNN-T ASR to speech disfluencies with a focus on partial words. For evaluation we use clean data, data with disfluencies and a separate dataset with speech affected by stuttering. We show that after including a small amount of data with disfluencies in the training set the recognition accuracy on the tests with disfluencies and stuttering improves. Increasing the amount of training data with disfluencies gives additional gains without degradation on the clean data. We also show that replacing partial words with a dedicated token helps to get even better accuracy on utterances with disfluencies and stutter. The evaluation of our best model shows 22.5% and 16.4% relative WER reduction on those two evaluation sets.

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

“…2.1.3 Dysfluent Speech Recognition. Technical work on improving speech assistants for PWS has focused on ASR models [8,23,31,35,50,51,61], stuttering detection [43], dysfluency detection or classification [22,40,42,48,56], clinical assessment [11], and dataset development [12,37,42,55]. Shonibare et al [61] and Mendelev et al [50] investigate training end-to-end RNN-T ASR models on speech from PWS.…”

“…Technical work on improving speech assistants for PWS has focused on ASR models [8,23,31,35,50,51,61], stuttering detection [43], dysfluency detection or classification [22,40,42,48,56], clinical assessment [11], and dataset development [12,37,42,55]. Shonibare et al [61] and Mendelev et al [50] investigate training end-to-end RNN-T ASR models on speech from PWS. Shonibare et al introduces a detect-then-pass approach that incorporates a dysfluency detector where audio frames with dysfluencies are ignored entirely by the RNN-T decoder.…”

“…Our approach has been to identify technical improvements that integrate with rather than replace existing systems, by offering relatively small adaptations to the speech recognition pipelinechanging an endpointer threshold, tuning the ASR's decoder model, and inserting a post-processing step to refine dysfluencies. Moreover, the techniques require very little data compared to what would be needed to train endpointer and ASR models from scratch, and are thus in contrast to many recent papers that replace existing systems with large ASR models trained on dysfluent speech (e.g., [2,50,61]) or personalized ASR models that require large amounts of data from a specific person (e.g., [33] which requires 15 -120 minutes of speech).…”

“…Research on speech technology for PWS has largely focused on technical improvements to automatic speech recognition (ASR) models [31,35,50,51,61], dysfluency detection [22,40,42,48], and dataset development [12,37,42,55]. This body of work has largely lacked a human-centered approach to understanding the experiences that PWS have with speech recognition systems [17], which could in turn inform how to prioritize and advance technical improvements.…”