Automated metrics for speech translation

Abstract

In this paper, we describe automated measures used to evaluate machine translation quality in the Defense Advanced Research Projects Agency"s Spoken Language Communication and Translation System for Tactical Use program, which is developing speech translation systems for dialogue between English and Iraqi Arabic speakers in military contexts. Limitations of the automated measures are illustrated along with variants of the measures that seek to overcome those limitations. Both the dialogue structure of the data and the Iraqi Arabic language challenge these measures, and the paper presents some solutions adopted by MITRE and NIST to improve confidence in the scores.

Machine translation for Arabic dialects (survey)

“…Several prototype systems were developed for military and medical screening domains to enable conversations with local foreign language speakers of Iraqi Arabic, Mandarin, Farsi, Pashto, and Thai. Some research was dedicated to evaluate MT scores of Iraqi Arabic and English translators such as (Condon et al, 2010) and (Condon et al, 2008). In the same context, IBM MASTOR (Gao et al, 2006), is a speech-to-speech translation system that translates spontaneous free-form speech in real-time on both laptop and hand-held PDAs for two language pairs, English-Mandarin Chinese, and English-Arabic dialect.…”