Leveraging a Character, Word and Prosody Triplet for an ASR Error Robust and Agglutination Friendly Punctuation Approach

Abstract

Punctuating ASR transcript has received increasing attention recently, and well-performing approaches were presented based on sequence-to-sequence modelling, exploiting textual (word and character) and/or acoustic-prosodic features. In this work we propose to consider character, word and prosody based features all at once to provide a robust and highly language independent platform for punctuation recovery, which can deal also well with highly agglutinating languages with less constrained word order. We demonstrate that using such a feature triplet improves ASR error robustness of punctuation in two quite differently organized languages, English and Hungarian. Moreover, in the highly agglutinating Hungarian, where word-based approaches suffer from the exploding vocabulary (poorer semantic representation through embeddings) and less constrained word order, we show that prosodic cues and the character-based model can powerfully counteract this loss of information. We also perform a deep analysis of punctuation w.r.t. both ASR errors and agglutination to explain the improvements we observed on a solid basis.

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

“…To address this issue, most of the earlier efforts on the punctuation restoration task have been done using lexical, acoustic, prosodic, or a combination of these features (Gravano et al, 2009;Levy et al, 2012;Zhang et al, 2013;Xu et al, 2014;Szaszák and Tündik, 2019;Che et al, 2016a). For the punctuation restoration task, lexical features have been widely used because the model can be trained using any punctuated text (i.e., publicly available newspaper articles or content from Wikipedia) and because of the availability of such large-scale text.…”

A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models

“…Most of the earlier work on punctuation restoration has been done using lexical, acoustic, and prosodic features or a combination of these features [41,83,130,186,195,198]. Lexical features are widely used for the task as the model can be trained with any well-punctuated text that is readily available, e.g., newspaper articles, Wikipedia, etc.…”

Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

“…This includes those that incorporate acoustic information (B. Garg and Anika, 2018;Moro and Szaszak, 2017;Szaszák and Tündik, 2019;Nanchen and Garner, 2019;Moró and Szaszák, 2017;Klejch et al, 2016Klejch et al, , 2017 and those that apply attention on top (Tilk and Alumäe, 2016;Oktem et al, 2017;Kim, 2019;Juin et al, 2017).…”