Extracting clues from human interpreter speech for spoken language translation

Abstract

In previous work, we reported dramatic improvements in automatic speech recognition (ASR) and spoken language translation (SLT) gained by applying information extracted from spoken human interpretations. These interpretations were artificially created by collecting read sentences from a clean parallel text corpus. Real human interpretations are significantly different. They suffer from frequent synopses, omissions and self-corrections. Expressing these differences in BLEU score by evaluating human interpretations with carefully created human translations, we found that human interpretations perform two to three times worse than state-of-the art SLT. Facing these stark differences, we address the question if and how ASR and SLT can profit from human interpretations. In the following we describe initial experiments that apply knowledge derived from real human interpretations for improving English and Spanish ASR and SLT. Our experiments are conducted on a small European Parliamentary Plenary Sessions development set.

Improving ASR by integrating lecture audio and slides

“…One can combine speech with a text stream, usually for an application such as machine-aided human translation [1,2], in which a human translator dictates the translation, rather than typing it. Also, a few works have looked into combining several speech streams [3,4], to improve ASR and MT systems in a simultaneous or consecutive interpretation scenario.…”

Simultaneous translation of lectures and speeches

“…We used ASR hypotheses as well as reference transcripts for the experiments, whereas the Spanish hypotheses were generated with a system trained within TC-STAR on Parliament plenary sessions (Stüker et al 2007;Paulik and Waibel 2008). The case-insensitive WER was 8.4%.…”

Spoken language translation from parallel speech audio: Simultaneous interpretation as SLT training data

“…Further, the hypotheses of both ASR systems can be tied together in a parallel training corpus suitable for TM training, as shown in [1]. Similar to previous works [2,3,4], we exploit the parallel information given in the respective other language audio stream to bias the ASR systems for an improved transcription performance. In the proposed context, such an improved ASR performance directly affects the quality of the extracted training data.…”