Listening while speaking: Speech chain by deep learning

Abstract

Despite the close relationship between speech perception and production, research in automatic speech recognition (ASR) and text-to-speech synthesis (TTS) has progressed more or less independently without exerting much mutual influence on each other. In human communication, on the other hand, a closed-loop speech chain mechanism with auditory feedback from the speaker's mouth to her ear is crucial. In this paper, we take a step further and develop a closed-loop speech chain model based on deep learning. The sequence-to-sequence model in close-loop architecture allows us to train our model on the concatenation of both labeled and unlabeled data. While ASR transcribes the unlabeled speech features, TTS attempts to reconstruct the original speech waveform based on the text from ASR. In the opposite direction, ASR also attempts to reconstruct the original text transcription given the synthesized speech. To the best of our knowledge, this is the first deep learning model that integrates human speech perception and production behaviors. Our experimental results show that the proposed approach significantly improved the performance more than separate systems that were only trained with labeled data.

Speech Recognition with Augmented Synthesized Speech

“…The term, speech chain, was first introduced in 1963 (and reissued in 1993) by Denes et al [8,9], where speech communication was described as a spoken message between the minds of the speakers and the listeners, based on speech production and speech perception processes. Using this as a basis, Tjandra et al, developed DeepChain [10], an approach to simultaneously train both recognition and synthesis systems. They proposed a sequence-to-sequence model in a closed-loop architecture that allows training with both, labeled and unlabeled data.…”

Speech-to-Speech Translation Between Untranscribed Unknown Languages

“…Even though the lack of natural speech dataset in this paper, VQ-VAE and codebook inverter can be applied and has shown a great performance on multispeaker natural speech [14,13]. Some papers [30,31,32] also show the performance improvement from the synthetic dataset can be carried over to the real dataset.…”

Cycle-consistency Training for End-to-end Speech Recognition

“…The speech chain model [9] is the most similar architecture to ours. As described in Section 1, the ASR model is trained with synthesized speech and the TTS model is trained with ASR hypotheses for unpaired data.…”