NMT-Based Segmentation and Punctuation Insertion for Real-Time Spoken Language Translation

Abstract

Insertion of proper segmentation and punctuation into an ASR transcript is crucial not only for the performance of subsequent applications but also for the readability of the text. In a simultaneous spoken language translation system, the segmentation model has to fulfill real-time constraints and minimize latency as well. In this paper, we show the successful integration of an attentional encoder-decoder-based segmentation and punctuation insertion model into a real-time spoken language translation system. The proposed technique can be easily integrated into the real-time framework and improve the punctuation performance on reference transcripts as well as on ASR outputs. Compared to the conventional language model and prosody-based model, our experiments on end-to-end spoken language translation show that translation performance is improved by 1.3 BLEU points by adopting the NMT-based punctuation model, maintaining low-latency.

Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging

“…The ones with plain text outperform the ones with To explore the impact of min_words_cut value to the quality of the result, we performed the experiment on sequenceto-sequence LSTM model with the overlapping of 15 words and min_words_cut ranges from 0 to 15. The outcome shown in Figure 5 indicates that f1-scores peak in the middle range of chunk size (4)(5)(6)(7)(8)(9)(10). It demonstrate that predictions of uppercase and lowercase are stable and independent from min_words_cut.…”

“…An example is shown in Figure 4. We prepared 2 formats of training data: plain text and encoded text [9]. Both formats takes the lowercase text without punctuation as input.…”

“…As we can see, with the help of Figure 5: F1-score on different min word cut. It peak in the middle range of overlap size (4)(5)(6)(7)(8)(9)(10). Predicting uppercase and lowercase are stable and independent from min word cut, question mark is quite sensitive with this hyper-parameter.…”

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

“…Previous work studied different decoding strategies to reduce the latency for real-time spoken language processing systems, including overlapping windows [7], streaming input scheme [8], and overlapped-chunk split and merging strategy [9]. However, the input text for inference in these decoding strategies does not always begin with the first word of a sentence.…”

Attentional Parallel RNNs for Generating Punctuation in Transcribed Speech

“…Punctuation is essential for grammaticality, readability, and (in the case of a number of different tasks), subsequent processing. Thus, correct sentence segmentation and punctuation of recognized speech improves the quality of machine translation [6,7,24,26], and missing periods and commas in machine generated text results in suboptimal information extraction from speech [13,15]. Also, most of the data-driven parsing models use punctuation as features.…”