Super-Human Performance in Online Low-latency Recognition of Conversational Speech

Abstract

Achieving super-human performance in recognizing human speech has been a goal for several decades, as researchers have worked on increasingly challenging tasks. In the 1990's it was discovered, that conversational speech between two humans turns out to be considerably more difficult than read speech as hesitations, disfluencies, false starts and sloppy articulation complicate acoustic processing and require robust handling of acoustic, lexical and language context, jointly. Early attempts with statistical models could only reach error rates over 50% and far from human performance (WER of around 5.5%). Neural hybrid models and recent attentionbased encoder-decoder models have considerably improved performance as such contexts can now be learned in an integral fashion. However, processing such contexts requires an entire utterance presentation and thus introduces unwanted delays before a recognition result can be output. In this paper, we address performance as well as latency. We present results for a system that can achieve super-human performance (at a WER of 5.0%, over the Switchboard conversational benchmark) at a word based latency of only 1 second behind a speaker's speech. The system uses multiple attention-based encoder-decoder networks integrated within a novel low latency incremental inference approach.

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

“…Attention-based models based on sequence-to-sequence (S2S) [3,37,40,49] are currently one of the top-performing approaches to end-to-end ASR and MT. A significant amount of study has already been spent to improving the performance of S2S models.…”

“…LSTMbased [39] models include 6 bidirectional layers for the encoder and 2 unidirectional layers for the decoder, with 1536 units in each. They have delivered superior recognition performance on the Switchboard conversational speech benchmark task [40]. The Transformer-based model proposed in [48] feature 24 encoder layers and 8 decoder layers.…”

“…individual sentences are translated and the output is commonly synthesized in a target language. Simultaneous interpretation of lectures, movies and video conferences by contrast are best delivered by subtitling [36,64], as they can be generated simultaneously [2,38,40,41,46] and do not create distractions during a speech or monologue. Still, when movies or offline video recordings are to be produced, subtitling is sometimes tiresome and a distraction of its own.…”