How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

Abstract

Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with stateof-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.

A Categorical Archive of ChatGPT Failures

“…• In this analysis, I took a high-level approach to examining the failures of ChatGPT. However, for future investigations, it may be useful to focus on more specific categories of problems, such as sentiment analysis, named entity recognition, translation [27,28], summarization [66], and language ambiguity [47], in order to gain a more detailed understanding of ChatGPT's shortcomings in these areas [49].…”

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

“…Meta's No Language Left Behind translates 200 different languages with high-quality results (Meta, 2022), and Google Translate, as of 2022, supports 133 languages, including 24 low-resource languages (Bapna, 2022). OpenAI's GPT models also emerge as excellent translators by generating context-relevant translation (Hendy et al, 2023). The approach proposed by Jung et al (2023) currently utilizes Google Translate API as well as GPT 3.5, which was shown to be capable of translating student responses with These examples demonstrate the impacts of LLMs and generative AI on automated scoring.…”

Exploring the relationship between critical thinking, attitude, and anxiety in shaping the adoption of artificial intelligence in translation among Saudi translators

“…The study also highlighted that GPT-3 achieved a relatively high level of accuracy in translating specialized religious text, with scores comparable to human translations in certain instances. Furthermore, Hendy et al (2023) evaluated GPT in the context of machine translation, exploring various dimensions such as the quality of different GPT models compared to state-of-the-art research and commercial systems, the impact of prompting strategies, robustness in the face of domain shifts, and document-level translation. The results suggested that GPT models demonstrated competitive translation quality for languages with ample resources but had limited capabilities when dealing with languages with scarce resources.…”