A preliminary evaluation of metadata records machine translation

Abstract

Purpose of this paperThe purpose of this study is to evaluate freely available machine translation (MT) services' performance in translating metadata records. Design/methodology/approachRandomly selected metadata records were translated from English into Chinese using Google, Bing, and SYSTRAN Machine Translation (MT) systems. These translations were then evaluated using a five point scale for both Fluency and Adequacy. Missing Count (words not translated) and Incorrect Count (words incorrectly translated) were also recorded. FindingsConcerning both Fluency and Adequacy, Google and Bing's translations of more than 70% of test data received scores equal to or greater than three, representative of 'non-native Chinese' and 'much coverage,' respectively. SYSTRAN scored lowest in both measures. However, these differences were not statistically significant. A Pearson correlation analysis demonstrated a strong relationship (r= .86) between Fluency and Adequacy. Missing Count and Incorrect Count strongly correlated with Fluency and Adequacy. Research limitations/implicationsThis study was conducted in a specific domain with a small sample size. It is necessary to conduct the evaluation with a larger, more representative test dataset. Also, other language pairs should be evaluated applying similar technologies. Originality/valueMost existing digital collections can be accessed in English alone. Few digital collections in the United States support multilingual information access (MLIA) that enables users of differing languages to search, browse, recognize and use information in the collections. Human translation is one solution, but it is neither time nor cost effective for most libraries. This study serves as a first step to understand the 2 performance of current MT systems and to design effective and efficient MLIA services for digital collections.

Metadata records machine translation combining multi‐engine outputs with limited parallel data

“…This study was one of the very few studies that conducted human evaluation of MT on metadata records. It confirmed the conclusion from the previous study (Chen et al, ) that online MT systems could produce non‐native yet sufficiently good translations that might help information users in many ways overcome language barriers. However, it significantly extended the MT evaluation in our previous study (Chen et al, ) to more languages and a larger sample size.…”

“…It confirmed the conclusion from the previous study (Chen et al, ) that online MT systems could produce non‐native yet sufficiently good translations that might help information users in many ways overcome language barriers. However, it significantly extended the MT evaluation in our previous study (Chen et al, ) to more languages and a larger sample size. Especially, we conducted evaluation on different elements and found that some online MT systems, such as Google Translate and Bing Translator, produced translations with high levels of fluency and adequacy for certain metadata elements such as subject and creator.…”

“…We doubled the Microsoft Bing translation results in the TM, which was justified by a previous study finding that Microsoft Bing slightly outperformed Google Translate and performed much better than Yahoo! in adequacy and fluency (Chen et al, ), as well as the fact that Microsoft Bing's translations ranked highest of the three in terms of BLEU scores for Chinese (Papineni, Roukos, Ward, & Zhu, ; Chen et al, ). The LM consisted of only the MT systems' output in Chinese or Spanish.…”

“…While many online MT systems are available for use, they are not always sufficient for producing quality MT metadata records to facilitate MLIA in digital collections. Chen, Ding, Jiang, and Knudson () experimented with three online MT systems that translated metadata records and found that MT performance of these records could stand improvement.…”

Machine translation and global research: Towards improved machine translation literacy in the scholarly community. LynneBowker and Jairo B.Ciro. Bingley, UK: Emerald Publishing, 2019. 128 pp. $95.00 (hardcover). (ISBN 9781787567221)

“…The evaluation of different MT services or open source systems for translation of scholarly articles would be useful for international scholars. There is some related research on evaluating MT systems for translation of metadata in digital libraries that would be worth discussing in this context (Chen et al, ; Reyes Ayala et al, ). This gap is partially justified by the complexity of those topics and the limited research on the multilingual aspects of scholarly communication.…”