A Phonological Phrase Sequence Modelling Approach for Resource Efficient and Robust Real-Time Punctuation Recovery

Abstract

For the automatic punctuation of Automatic Speech Recognition (ASR) output, both prosodic and text based features are used, often in combination. Pure prosody based approaches usually have low computation needs, introduce little latency (delay) and they are also more robust to ASR errors. Text based approaches usually yield better performance, they are however resource demanding (both regarding their training and computational needs), often introduce high time latency and are more sensitive to ASR errors. The present paper proposes a lightweight prosody based punctuation approach following a new paradigm: we argue in favour of an all-inclusive modelling of speech prosody instead of just relying on distinct acoustic markers: first, the entire phonological phrase structure is reconstructed, then its close correlation with punctuations is exploited in a sequence modelling approach with recurrent neural networks. With this tiny and easy to implement model we reach performance in Hungarian punctuation comparable to large, text based models for other languages by keeping resource requirements minimal and suitable for real-time operation with low latency.

Leveraging a Character, Word and Prosody Triplet for an ASR Error Robust and Agglutination Friendly Punctuation Approach

“…Recently, we have addressed punctuation of Hungarian and English using text (both on word and character basis) [2,16] and prosodic-acoustic approaches [1]. We observed, that punctuation models for English [2,9,10,13] consistently work better for periods (full stops) than commas.…”

“…We observed, that punctuation models for English [2,9,10,13] consistently work better for periods (full stops) than commas. In contrast, for Hungarian, text-based punctuation works better for commas, whereas prosody yields better period recovery [1]. We believe that this is closely linked to some basic characteristics of the two languages (other than comma usage standards, which are quite similar), i.e.…”

“…In [1], the authors evaluated punctuation based on prosodic features derived from phonological phrase alignment. Since this approach involves a separate Hidden Markov-model to obtain the phonological phrase segmentation [19], we replace this with an end-to-end pipeline in this work, and let most of the modelling job be done by a neural network, but we keep the acoustic-prosodic feature pre-processing steps.…”

“…Punctuating the produced transcripts is an important add-on in Automatic Speech Recognition (ASR), which is not inherently handled by most state-of-the-art ASR systems. The problem is well-known, therefore we overview here only the aspects relevant for our contribution, and refer the reader to [1,2,3] for further exploration of the topic.…”

“…Both text and acoustics can be exploited for punctuation; text-based models [2,6,8,10,12] are more complex, but yield better results than models based on prosodic-acoustic features [1,7,14], which are more resistant to ASR errors. The two types of features are used in parallel in some studies [7,13,11] and most of the time, pause duration is used by text-based systems.…”