Replacing Human Audio with Synthetic Audio for on-Device Unspoken Punctuation Prediction

Soboleva, Daria; Skopek, Ondrej; Šajgalík, Márius; Carbune, Victor; Weissenberger, Felix; Proskurnia, Julia; Prisacari, Bogdan; Valcarce, Daniel; Lu, Justin; Prabhavalkar, Rohit; Bálint, Miklós

doi:10.1109/icassp39728.2021.9413432

Cited by 9 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through augmentation of authentic human datasets using TTS, not only does this enhance the accuracy of self-training processes [11], but also the generalizability of ASR models by introducing prosodic and acoustic variations [12,13] into training data. Furthermore, as recent TTS models [8,9] are able to produce more natural, human-like speech, this has prompted researchers to explore synthetic speech datasets as complete substitutes for authentic human speech datasets [14,15]. Nevertheless, despite these notable developments in other research areas, no prior investigations have explored the potential impact of using synthetic data in DST.…”

Section: Related Workmentioning

confidence: 99%

A Study on the Curriculum of the Graduate School of Department of Education Consulting in South Korea

Lee¹

2022

Korea Assoc Edu Consulting Coaching

View full text Add to dashboard Cite

The purpose of this study was to suggest implications for the curriculum of the graduate school of department of education consulting and education consulting major in South Korea through the analysis of the current status and curriculum of the graduate school of department of education consulting and education consulting major in South Korea. In this study, in order to investigate the current status and curriculum of the graduate school of department of education consulting and education consulting major in South Korea, the keywords were limited to ‘education consulting’, and data were collected in the order of Daehakalrimi(https://www.academyinfo.go.kr), Naver and Google keyword search, and each university and graduate school’s website. As a result of the analysis of the collected data, it was confirmed that three master’s and doctoral programs in general graduate schools in South Korea, and nine master’s programs in graduate schools of education in South Korea were in operation, and the curriculum of the graduate school of department of education consulting and education consulting major in South Korea were organized based on the faculty members’s major and expertise according to new department and major or name changed department and major. This study is the first study to analysis the current status and curriculum of the graduate school of department of education consulting and education consulting major in South Korea and is meaningful as a basic study of the development of education consulting as disciplines.

show abstract

Section: Related Workmentioning

confidence: 99%

A Study on the Curriculum of the Graduate School of Department of Education Consulting in South Korea

Lee¹

2022

Korea Assoc Edu Consulting Coaching

View full text Add to dashboard Cite

show abstract

“…We compare the performance of UniPunc with various baselines and SOTA systems, including: LSTM-T [13], Att-GRU [20], BERT [7], SAPR [8], Self-Att-Word-Speech [10] and MuSe [11]. We also compare to TTS punctuation data augmentation [19].…”

Section: Configurations and Baselinesmentioning

confidence: 99%

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Zhu¹,

Wu²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The punctuation restoration task aims to correctly punctuate the output transcriptions of automatic speech recognition systems. Previous punctuation models, either using text only or demanding the corresponding audio, tend to be constrained by real scenes, where unpunctuated sentences are a mixture of those with and without audio. This paper proposes a unified multimodal punctuation restoration framework, named UniPunc, to punctuate the mixed sentences with a single model. UniPunc jointly represents audio and non-audio samples in a shared latent space, based on which the model learns a hybrid representation and punctuates both kinds of samples. We validate the effectiveness of the UniPunc on real-world datasets, which outperforms various strong baselines (e.g. BERT, MuSe) by at least 0.8 overall F1 scores, making a new state-of-the-art. Extensive experiments show that UniPunc's design is a pervasive solution: by grafting onto previous models, UniPunc enables them to punctuate on the mixed corpus. Our code is available at github.com/Yaoming95/UniPunc

show abstract

“…Secondly, the Parrotron model is designed to produce a generic speech, while in our task, it is critical to retain the child's original voice. Similarly, algorithms for foreign accent conversion [6] and pronunciation prediction [7] will not meet our lack of data and the goal of maintaining the speaker voice.…”

Section: Introductionmentioning

confidence: 99%

Correcting Mispronunciations in Speech using Spectrogram Inpainting

Ben-Simon¹,

Kreuk²,

Awwad³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning a new language involves constantly comparing speech productions with reference productions from the environment. Early in speech acquisition, children make articulatory adjustments to match their caregivers' speech. Grownup learners of a language tweak their speech to match the tutor reference. This paper proposes a method to synthetically generate correct pronunciation feedback given incorrect production. Furthermore, our aim is to generate the corrected production while maintaining the speaker's original voice.The system prompts the user to pronounce a phrase. The speech is recorded, and the samples associated with the inaccurate phoneme are masked with zeros. This waveform serves as an input to a speech generator, implemented as a deep learning inpainting system with a U-net architecture, and trained to output a reconstructed speech. The training set is composed of unimpaired proper speech examples, and the generator is trained to reconstruct the original proper speech. We evaluated the performance of our system on phoneme replacement of minimal pair words of English as well as on children with pronunciation disorders. Results suggest that human listeners slightly prefer our generated speech over a smoothed replacement of the inaccurate phoneme with a production of a different speaker.

show abstract

Replacing Human Audio with Synthetic Audio for on-Device Unspoken Punctuation Prediction

Cited by 9 publications

References 17 publications

A Study on the Curriculum of the Graduate School of Department of Education Consulting in South Korea

A Study on the Curriculum of the Graduate School of Department of Education Consulting in South Korea

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Correcting Mispronunciations in Speech using Spectrogram Inpainting

Contact Info

Product

Resources

About