A Survey on Evaluation Metrics for Machine Translation

Lee, Seungjun; Lee, Jungseob; Moon, Hyeonseok; Park, Chanjun; Seo, Jaehyung; Eo, Sugyeong; Koo, Seonmin; Lim, Heuiseok

doi:10.3390/math11041006

Cited by 20 publications

(24 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recognizing the importance of evaluation methods is relevant to the study's assessment of attitudes toward TMS quality and performance. MT systems have applications in various domains, including the business and professional translation sectors (Lee et al ., 2023). Understanding these applications is fundamental to comprehending the role of TMS in the translation industry.…”

Section: Introductionmentioning

confidence: 99%

Attitudes of translation agencies and professional translators in Saudi Arabia towards translation management systems

Alsubhi

2024

SJLS

View full text Add to dashboard Cite

PurposeEffective translation has become essential for seamless cross-cultural communication in an era of global interconnectedness. Translation management systems (TMS) have redefined the translation landscape, revolutionizing project management and execution. This study examines the attitudes of translation agencies and professional translators towards integrating and utilizing TMS, with a specific focus on Saudi Arabia.Design/methodology/approachThe study's design was based on a thorough mixed-methods strategy that purposefully combined quantitative and qualitative procedures to create an array of findings. Through a survey involving 35 participants (both project managers and professional translators) and a series of interviews, this research explores the adoption of TMS, perceived benefits, influencing factors and future considerations. This integrated approach sought to investigate the nuanced perceptions of Saudi translation companies and expert translators about TMS. By combining the strengths of quantitative data's broad scopes and qualitative insights' depth, this mixed-methods approach sought to overcome the limitations of each method, ultimately resulting in a holistic understanding of the multifaceted factors shaping attitudes within Saudi Arabia's unique translation landscape.FindingsBased on questionnaires and interviews, the study shows that 80% of participants were familiar with TMS, and 57% had adopted it in their work. Benefits included enhanced project efficiency, collaboration and quality assurance. Factors influencing adoption encompassed cost, compatibility and resistance to change. The study further delved into participants' demographic profiles and years of experience, with a notable concentration in the 6–10 years range. TMS adoption was linked to improved translation processes, and participants expressed interest in AI integration and mobile compatibility. Deployment models favored cloud-based solutions, and compliance with industry standards was deemed vital. The findings underscore the evolving nature of TMS adoption in Saudi Arabia, with diverse attitudes shaped by cultural influences, technological compatibility and awareness.Originality/valueThis research provides a holistic and profound perspective on the integration of TMS, fostering a more comprehensive understanding of the opportunities, obstacles and potential pathways to success. As the translation landscape continues to evolve, the findings from this study will serve as a valuable compass guiding practitioners and researchers towards effectively harnessing the power of technology for enhanced translation outcomes.

show abstract

Section: Introductionmentioning

confidence: 99%

Attitudes of translation agencies and professional translators in Saudi Arabia towards translation management systems

Alsubhi

2024

SJLS

View full text Add to dashboard Cite

show abstract

“…Blur criteria and scales for manual translation quality, along with different human evaluator sensitivity to translation errors may result in the judge subjectivity, which can be reflected in the poor consistency and instability of the evaluation results 5 . Human evaluation is an effective way to assess translation quality, but is challenging to find reliable bilingual annotators 6 . In addition to poor consistency and subjectivity, manual evaluation is both financially and time-consuming; however, unlike automatic evaluation, it does not require a reference translation.…”

Section: Introductionmentioning

confidence: 99%

“…The advantages of automatic evaluation lie in its objectivity, consistency, stability, speed, reusability, and language independence. It is cost-effective and easy to use for comparing multiply systems, but at the expense of quality 6 . Furthermore, automatic evaluation requires reference—human translation (gold standard)—since the evaluation is based on comparing MT output with reference translation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The use of residual analysis to improve the error rate accuracy of machine translation

Benko,

Munkova,

Munk

et al. 2024

Sci Rep

View full text Add to dashboard Cite

The aim of the study is to compare two different approaches to machine translation—statistical and neural—using automatic MT metrics of error rate and residuals. We examined four available online MT systems (statistical Google Translate, neural Google Translate, and two European commission’s MT tools—statistical mt@ec and neural eTranslation) through their products (MT outputs). We propose using residual analysis to improve the accuracy of machine translation error rate. Residuals represent a new approach to comparing the quality of statistical and neural MT outputs. The study provides new insights into evaluating machine translation quality from English and German into Slovak through automatic error rate metrics. In the category of prediction and syntactic-semantic correlativeness, statistical MT showed a significantly higher error rate than neural MT. Conversely, in the category of lexical semantics, neural MT showed a significantly higher error rate than statistical MT. The results indicate that relying solely on the reference when determining MT quality is insufficient. However, when combined with residuals, it offers a more objective view of MT quality and facilitates the comparison of statistical MT and neural MT.

show abstract

“…BLEU scores can also be used to track language learners' progress over time as learners strive to produce text that is increasingly similar to humangenerated text. BLEU is relatively simple to compute and widely used, but it has been criticized for not considering the meaning of the translated text or the context in which it is used (Lee et al, 2023). Therefore, using multiple metrics is important to get a complete picture.…”

mentioning

confidence: 99%

Machine translation in language acquisition: A study on EFL students' perceptions and practices in Saudi Arabia and South Korea

Almusharraf

Bailey

2023

Computer Assisted Learning

View full text Add to dashboard Cite

BackgroundLanguage learners are taking advantage of the increased accuracy afforded by machine translation (MT) tools like Google Translate. There is a growing debate as to whether these tools support or hinder English as a foreign language (EFL) education. Some EFL students are drawn to MT tools to save time and energy when communicating, but how this use relates to second language acquisition needs to be clarified.ObjectiveThis study investigated the perceptions and practices of MT tools among university EFL students in Saudi Arabia and South Korea.MethodsThe study surveyed Saudi (n = 310) and South Korean (n = 160) university students to explore the use of MT tools in EFL learning. It used mean scores, correlation analysis, and structural equation modelling to understand the perceived benefits and limitations of MT tools and their influence on the use of translation websites. The findings were triangulated using content analysis of open‐ended items.Results and ConclusionFindings indicate students showed a high level of acceptance and utilization of MT tools. English avoidance behaviour (B = 0.43), benefits (B = 0.32), and foreign language proficiency (B = −0.19) predicted 61% of actual use of MT tools like Google Translate. Further, students frequently used MT tools despite the reported concerns about accuracy mistakes. Saudi students listed a more comprehensive array of why they use MT tools than Korean students, while Saudis and Koreans frequently used MT tools for individual word and sentence translations.ImplicationsStudents should edit MT output when writing and are encouraged to use the voice recording and other functions of MT tools to learn language skills. Instructors must prevent overuse and dependency on MT tools, especially among low‐proficiency students who want to avoid using English.

show abstract

A Survey on Evaluation Metrics for Machine Translation

Cited by 20 publications

References 33 publications

Attitudes of translation agencies and professional translators in Saudi Arabia towards translation management systems

Attitudes of translation agencies and professional translators in Saudi Arabia towards translation management systems

The use of residual analysis to improve the error rate accuracy of machine translation

Machine translation in language acquisition: A study on EFL students' perceptions and practices in Saudi Arabia and South Korea

Contact Info

Product

Resources

About