Multimodal Punctuation Prediction with Contextual Dropout

Abstract

Automatic speech recognition (ASR) is widely used in consumer electronics. ASR greatly improves the utility and accessibility of technology, but usually the output is only word sequences without punctuation. This can result in ambiguity in inferring user-intent. We first present a transformerbased approach for punctuation prediction that achieves 8% improvement on the IWSLT 2012 TED Task, beating the previous state of the art [1]. We next describe our multimodal model that learns from both text and audio, which achieves 8% improvement over the text-only algorithm on an internal dataset for which we have both the audio and transcriptions. Finally, we present an approach to learning a model using contextual dropout that allows us to handle variable amounts of future context at test time.

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

“…For the previously proposed punctuation model, a proper adaptation of the proposed coordinate bootstrapper and acoustic assistant can empower them to address modality missing samples. Section 5 discuss the the pervasiveness of coordinate bootstrapper upon previous models [5,12] via experiment.…”

“…This subsection explores the pervasiveness of UniPunc framework for punctuation restoration on the mixed datasets by grafting onto two previous approaches, namely BiLSTM [5] and content dropout [12]. Specifically, for the BiLSTM model, we introduce an acoustic assistant to extract potential acoustic features and coordinate bootstrapper to learn hybrid representation.…”

“…b) UniPunc achieves state-of-theart performance on multimodal punctuation and outperforms various strong baselines by at least 0.8% overall F1 scores on real-world datasets. c) We discuss pervasiveness UniPunc: the introduced framework can be grafted into related work [5,12], empowering them to process both audio and non-audio text while improving performance.…”