Restoring Punctuation and Capitalization Using Transformer Models

Vāravs, Andris; Salimbajevs, Askars

doi:10.1007/978-3-030-00810-9_9

Cited by 16 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 2 illustrates the sequence-to-sequence encoder-decoder architecture used in this paper. We note that a couple of previous works used machine translation approaches to address punctuation restoration (Peltz et al 2011, Vāravs andSalimbajevs 2018). Similar to this work, these past works frame the punctuation restoration task as a text-to-text task.…”

Section: Related Workmentioning

confidence: 70%

Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Melamud

Duran

2021

J Stud Res

View full text Add to dashboard Cite

When creating text transcripts from spoken audio, Automatic Speech Recognition (ASR) systems need to infer appropriate punctuation in order to make the transcription more readable. This task, known as punctuation restoration, is challenging since punctuation is not explicitly stated in speech. Most recent works framed punctuation restoration as a classification task and used pre-trained encoder-based transformers, such as BERT, to perform it. In this work, we present an alternative approach, framing punctuation restoration as a sequence-to-sequence task and using T5, a pretrained encoder-decoder transformer model, as the basis of our implementation. Training our model on IWSLT 2012, a common punctuation restoration benchmark, we find its performance is comparable to state of the art classification-based systems with an F1 score of 80.7 on the test set. Furthermore, we argue that our approach might be more flexible in its ability to adapt to more complex types of outputs, such as predicting more than one punctuation mark in a row.

show abstract

Section: Related Workmentioning

confidence: 70%

Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Melamud

Duran

2021

J Stud Res

View full text Add to dashboard Cite

show abstract

“…Garg and Anika, 2018;Che et al, 2016;Żelasko et al, 2018)), which have less predictive power. The handful of approaches that make use of Transformer architectures are not bidirectional Nguyen et al, 2019;Vāravs and Salimbajevs, 2018;Wang et al, 2018). Our model also differs from the above in that it leverages pretraining to reduce training time and increase accuracy.…”

Section: Related Workmentioning

confidence: 99%

Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

Courtland¹,

Faulkner²,

McElvain³

2020

Proceedings of the 17th International Conference on Spoken Language Translation

View full text Add to dashboard Cite

Though people rarely speak in complete sentences, punctuation confers many benefits to the readers of transcribed speech. Unfortunately, most ASR systems do not produce punctuated output. To address this, we propose a solution for automatic punctuation that is both cost efficient and easy to train. Our solution benefits from the recent trend in fine-tuning transformer-based language models. We also modify the typical framing of this task by predicting punctuation for sequences rather than individual tokens, which makes for more efficient training and inference. Finally, we find that aggregating predictions across multiple context windows improves accuracy even further. Our best model achieves a new state of the art on benchmark data (TED Talks) with a combined F1 of 83.9, representing a 48.7% relative improvement (15.3 absolute) over the previous state of the art.

show abstract

“…Garg and Anika, 2018;Che et al, 2016;Żelasko et al, 2018)), which have less predictive power. The handful of approaches that make use of Transformer architectures are not bidirectional Vāravs and Salimbajevs, 2018;. Our model also differs from the above in that it leverages pretraining to reduce training time and increase accuracy.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 17th International Conference on Spoken Language Translation

Federico¹,

Waibel²,

Knight³

et al. 2020

View full text Add to dashboard Cite

The conference chairs and organizers would like to express their gratitude to everyone who contributed and supported IWSLT. Our IWSLT-20 program exceeds all our expectations in quality and breath, particularly when considering the challenges during a pandemic under lock-downs and health and travel restrictions. We thank the challenge track chairs, organizers, and participants, the program chairs and committee members, as well as all the authors that went the extra mile to submit system and research papers to IWSLT, and make this year's conference our most vibrant than ever. We also wish to express our sincere gratitude to ACL for hosting our conference and for arranging the logistics and infrastructure that allow us to hold IWSLT 2020 as a virtual online conference.

show abstract

Restoring Punctuation and Capitalization Using Transformer Models

Cited by 16 publications

References 15 publications

Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Punctuation Restoration for Speech Transcripts using seq2seq Transformers

Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

Proceedings of the 17th International Conference on Spoken Language Translation

Contact Info

Product

Resources

About