How do measures of cognitive effort relate to each other? A multivariate analysis of post-editing process data

Abstract

There has been growing interest of late in the cognitive effort required by post-editing of machine translation. Compared to number of editing operations, cognitive (or mental) effort is frequently considered a more decisive indicator of the overall effort expended by post-editors. Estimating cognitive effort is not straightforward, however. Previous studies often triangulate different measures to obtain a consensus, but little post-editing research to date has attempted to show how measures of cognitive effort relate to each other in a multivariate analysis. This paper addresses this by presenting an exploratory comparison of cognitive measures based on eye tracking, pauses, editing time, and subjective ratings collected in a post-editing task carried out by professional and non-professional participants. All measures correlated with each other, but a principal components analysis showed that the measures cluster together in different ways. In particular, measures that increase with task time alone behaved differently from the others, with higher mutual associations and higher reliability. Regarding differences between professional and non-professional participants, it was observed that subjective ratings were overall more strongly associated with objective measures in the case of professionals. Surprising findings from previous research based on pause ratio are discussed. The paper argues that a pause typology will benefit the study of pause lengths and cognitive effort in post-editing.

A Comparative Study on the Effort of Human Translation and Post-Editing in Relation to Text Types: An Eye-Tracking and Key-Logging Experiment

“…ST features influence MT quality and PE effort (Krings, 2001, p. 58; Vieira, 2017, p. 42). In this study, we mainly considered textual functions and linguistic features when selecting experiment materials, and we chose four text types with distinctive features, namely, advertising, legal, news, and poetic texts.…”

“…Considering that translation involves reading, which is the basis for transferring between languages, and fixations in reading last from 100 ms to over 500 ms (Pavlović & Jensen, 2009, p. 97), we set the threshold at 100 ms and deleted the fixations lasting less than 100 ms. Besides, we calculated the Fixation Duration per Word (FDW) as a complement to FCW, because fixation duration could also reveal participants’ cognitive effort (Castilho, 2016, p. 43), and the measures of fixation calculated with reference to word count are more reliable (Vieira, 2017). We measured FCW and FDW for both ST and TT, which were calculated with the number of significant fixations and the total fixation duration divided by word count, namely, Source Fixation Duration per Source Word (SFDW), Target Fixation Duration per Target Word (TFDW), Source Fixation Count per Source Word (SFCW), and Target Fixation Count per Target Word (TFCW).…”

Post-editing Effort of a Novel With Statistical and Neural Machine Translation

“…The results in terms of time and quality guide the conception of a new subtitling tool. For the analysis of effort, we use established measures based on gaze and typing data, and subjective ratings (de Sousa, Aziz & Specia, 2011;Vieira, 2016). Our hypotheses were that post-editing is faster than translation tasks from scratch and that access to the video is essential for the post-editing task even if the source language is unknown.…”

Dictating translations with automatic speech recognition: Effects on translators’ performance

“…3 https://sites.google.com/site/centretranslationinnovation/tpr-db Pauses are often considered a reliable indicator of cognitive effort in language production and translation processes (Schilperoord, 1996;Lacruz and Shreve, 2014;Vieira, 2016;Lacruz, 2017). However, there is disagreement on the appropriate threshold for pauses to accurately reflect translation effort (Muñoz Martín and Cardona, 2019).…”