Analysis and Tuning of a Voice Assistant System for Dysfluent Speech

Mitra, Vikramjit; Huang, Zifang; Tooley, Lauren; Wu, Sarah; Botten, Darren; Palekar, Ashwini; Thelapurath, Shrinath; Georgiou, Panayiotis G.; Kajarekar, Sachin; Bigham, Jeffrey P.

doi:10.21437/interspeech.2021-2006

Cited by 14 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Visualization: Figure. 4 shows the embeddings computed from the statistics pooling layer. In MTL scheme, it is evident from the well formed podcast clusters that the model is trying to learn podcast dependent stuttering information.…”

Section: Resultsmentioning

confidence: 99%

“…However, this method of detecting stuttering is very arduous and time-intensive, and is also prejudiced towards the subjective belief of speech therapist. Besides, automatic speech recognition (ASR) tools also fail to recognize the stuttered speech [4], that makes it unrealistic to easily access virtual assistants like Alexa, Apple Siri, Cortana, etc. for PWS.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Robust Stuttering Detection via Multi-task and Adversarial Learning

Sheikh¹,

Sahidullah²,

Hirsch³

et al. 2022

Preprint

View full text Add to dashboard Cite

By automatic detection and identification of stuttering, speech pathologists can track the progression of disfluencies of persons who stutter (PWS). In this paper, we investigate the impact of multi-task (MTL) and adversarial learning (ADV) to learn robust stutter features. This is the first-ever preliminary study where MTL and ADV have been employed in stuttering identification (SI). We evaluate our system on the SEP-28k stuttering dataset consisting of ≈ 20 hours of data from 385 podcasts. Our methods show promising results and outperform the baseline in various disfluency classes. We achieve up to 10%, 6.78%, and 2% improvement in repetitions, blocks, and interjections respectively over the baseline.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Robust Stuttering Detection via Multi-task and Adversarial Learning

Sheikh¹,

Sahidullah²,

Hirsch³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This convention of SD is very laborious and timeconsuming and is also inclined toward the idiosyncratic belief of speech therapists. In addition, the automatic speech recognition systems (ASR) are working fine for normal fluent speech, however, they are unsuccessful in recognizing the stuttered speech [9], which makes it impractical for PWS to easily access virtual assistants like Apple Siri, Alexa, etc. As a result, interactive automatic SD systems that provide an impartial objective, and consistent evaluation of stuttering speech are strongly encouraged.…”

Section: Introductionmentioning

confidence: 99%

“…Fig.3: A schematic diagram of Multi-contextual StutterNet, which is a multi-class classifier that exploits different variable contexts of(5,9) in SD. The FluentBranch and DisfluentBranch are composed of 3 fully connected layers followed by a softmax layer for prediction of different stuttering classes, CB: Context Block, SPL: Statistical Pooling Layer.…”

mentioning

confidence: 99%

Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning

Sheikh

Sahidullah

Hirsch³

et al. 2023

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score (F 1 ). In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in F 1 over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F 1 over the clean training.

show abstract

“…(e.g. with voice assistants or speech dictation systems) [3], [25], [26], [31]. With these applications in mind, we present new techniques for automatic disfluency detection, categorization, and localization.…”

Section: Introductionmentioning

confidence: 99%

Toward A Multimodal Approach for Disfluency Detection and Categorization

Romana

2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Speech disfluencies, such as filled pauses or repetitions, are disruptions in the typical flow of speech. Stuttering is a speech disorder characterized by a high rate of disfluencies, but all individuals speak with some disfluencies and the rates of disfluencies may by increased by factors such as cognitive load. Clinically, automatic disfluency detection may help in treatment planning for individuals who stutter. Outside of the clinic, automatic disfluency detection may serve as a pre-processing step to improve natural language understanding in downstream applications. With this wide range of applications in mind, we investigate language, acoustic, and multimodal methods for frame-level automatic disfluency detection and categorization. Each of these methods relies on audio as an input. First, we evaluate several automatic speech recognition (ASR) systems in terms of their ability to transcribe disfluencies, measured using disfluency error rates. We then use these ASR transcripts as input to a language-based disfluency detection model. We find that disfluency detection performance is largely limited by the quality of transcripts and alignments. We find that an acoustic-based approach that does not require transcription as an intermediate step outperforms the ASR language approach. Finally, we present multimodal architectures which we find improve disfluency detection performance over the unimodal approaches. Ultimately, this work introduces novel approaches for automatic frame-level disfluency and categorization. In the long term, this will help researchers incorporate automatic disfluency detection into a range of applications.

show abstract

Analysis and Tuning of a Voice Assistant System for Dysfluent Speech

Cited by 14 publications

References 13 publications

Robust Stuttering Detection via Multi-task and Adversarial Learning

Robust Stuttering Detection via Multi-task and Adversarial Learning

Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning

Toward A Multimodal Approach for Disfluency Detection and Categorization

Contact Info

Product

Resources

About