The impact of translation modality on user experience: an eye-tracking study of the Microsoft Word user interface

Abstract

This paper presents results of the effect of different translation modalities on users when working with the Microsoft Word user interface. An experimental study was set up with 84 Japanese, German, Spanish, and English native speakers working with Microsoft Word in three modalities: the published translated version, a machine translated (MT) version (with unedited MT strings incorporated into the MS Word interface) and the published English version. An eye-tracker measured the cognitive load and usability according to the ISO/TR 16982 guidelines: i.e., effectiveness, efficiency, and satisfaction followed by retrospective think-aloud protocol. The results show that the users’ effectiveness (number of tasks completed) does not significantly differ due to the translation modality. However, their efficiency (time for task completion) and self-reported satisfaction are significantly higher when working with the released product as opposed to the unedited MT version, especially when participants are less experienced. The eye-tracking results show that users experience a higher cognitive load when working with MT and with the human-translated versions as opposed to the English original. The results suggest that language and translation modality play a significant role in the usability of software products whether users complete the given tasks or not and even if they are unaware that MT was used to translate the interface.

Is machine translation a dim technology for its users? An eye tracking study

“…In a study by Guerberof Arenas et al ( 2021 ), researching the effect of different translation modalities on users through an eye tracking experiment, 79 end users' (Japanese, German, Spanish, English) experiences with published translated, machine-translated and published English versions were compared. The authors focused on the number of successful tasks performed by end users, the time necessary for performing successful tasks in different translation modalities, the satisfaction level of end users in relation to different translation modalities and the amount of cognitive effort necessary for carrying out tasks in different translation modalities (Guerberof Arenas et al, 2021 ). They measured usability, i.e., effectiveness (by asking participants to perform some tasks), efficiency (by measuring the time to complete the tasks and by measuring cognitive effort using an eye tracker) and satisfaction.…”

“…The results of the eye tracking experiment revealed that end users' cognitive load was higher for machine-translated and human translated versions than for the English original. The findings also indicated that the language and the translation modality played a significant role in the usability, regardless of whether end users finished the given tasks and even if they were unaware that MT was used (Guerberof Arenas et al, 2021 ).…”

Audiovisual translation, translators, and technology: From automation pipe dream to human–machine convergence

“…However, the concern still exists regarding the complete automation of translation processes because of its negative effects on professional working conditions (O'Hagan, 2019; Vieira, 2018). Some higher education programs have already begun providing specific training in MT and post-editing (Cid-Leal et al, 2019;Guerberof Arenas & Moorkens, 2019;O'Brien & Salis, 2002;Rico Pérez, 2017;Venkatesan, 2018) or have undertaken innovations on a smaller scale (González Pastor, 2021;González Pastor & Rico, 2021;Mejías-Climent & de los Reyes Lozano, 2021;Moorkens, 2018;).…”

Ethics and Machine Translation: The End User Perspective

“…the mental effort employed in completing (or not) the tasks. For a detailed methodology of this experiment, refer to Guerberof-Arenas et al (2019.…”