Factors Influencing the Use of Captions by Foreign Language Learners: An Eye‐Tracking Study

Abstract

This study investigates caption‐reading behavior by foreign language (L2) learners and, through eye‐tracking methodology, explores the extent to which the relationship between the native and target language affects that behavior. Second‐year (4th semester) English‐speaking learners of Arabic, Chinese, Russian, and Spanish watched 2 videos differing in content familiarity, each dubbed and captioned in the target language. Results indicated that time spent on captions differed significantly by language: Arabic learners spent more time on captions than learners of Spanish and Russian. A significant interaction between language and content familiarity occurred: Chinese learners spent less time on captions in the unfamiliar content video than the familiar, while others spent comparable times on each. Based on dual‐processing and cognitive load theories, we posit that the Chinese learners experienced a split‐attention effect when verbal processing was difficult and that, overall, captioning benefits during the 4th semester of language learning are constrained by L2 differences, including differences in script, vocabulary knowledge, concomitant L2 proficiency, and instructional methods. Results are triangulated with qualitative findings from interviews.

Effects of Subtitles, Complexity, and Language Proficiency on Learning from Online Education Videos

“…These null findings contradict two lines of research, one showing beneficial effects of subtitles, the other showing detrimental effects. Earlier research which has shown beneficial effects of subtitles are primarily studies on second language learning, which show that 2 nd language subtitles help students with learning that language (e.g., Baltova, 1999;Chung, 1999;Markham, 1999;Winke et al, 2013). While this appears conflicting with the results of the present study, the important difference is that the current study did not use language learning videos but 'content' videos, and did not measure gains in 2 nd language proficiency.…”

“…Specifically, enabling subtitles for language learning videos substantially increases student performance on recognition tests, and to a lesser extent on production tests. Other studies have found similar positive effects of subtitles on learning from videos and there appears to be a consensus that subtitles are beneficial for learning a second language (e.g., Baltova, 1999;Chung, 1999;Markham, 1999;Winke, Gass, & Sydorenko, 2013). However, these are all studies which focus on learning a language and not on learning about a non-linguistic topic in a second language.…”

Intralingual subtitles, interlingual subtitles, and video comprehension: insights from an exploratory study

“…Test announcement has indeed been found to correlate with participants' better performance on tests. Winke, Gass, and Sydorenko (2013) found that their participants in the test announcement condition using full captioning or keyword captioning performed better than participants who did not receive a test announcement before watching the video in those conditions. The authors posit that test announcement possibly prompted them to reanalyze the content of the subtitles more closely, thus spending additional time on the subtitles area to process it more effortfully and therefore gather more information to understand the narration.…”

“…Markham, Peter, and McCarthy's (2001) revealed that participants' level of recall of the video content was much lower when no subtitles were available than when they were for non-familiar material, that is, for video materials with which participants were not so familiar in terms of content/genre. Also, Winke, Gass, and Sydorenko's (2013) eye-tracking study demonstrated that video content familiarity correlated with the type of L2 being learned, which, in that case, was Chinese, a language that required more processing of subtitles with a video that was deemed more difficult for the kind of topic it portrayed, being unfamiliar to the participants.…”

“…Since reading subtitles has been found to be an automatic behavior on the viewers' part regardless of one's familiarity with them, knowledge of the foreign language in the soundtrack or its availability -as confirmed in studies with eye-movement recordings (D'YDEWALLE & GIELEN, 1992;WINKE, GASS, & SYDORENKO, 2013;KRUGER & STEYN, 2014;MONTERO-PEREZ, PETERS, & DESMET, 2015) -it then seems reasonable to assume that part of one's attentional resources when watching a subtitled film is allocated in reading and processing the subtitles, whereas the rest is ideally distributed between the processes of watching and attending to the story, processing the motion picture, and arguably attending to the auditory input. Hence, the ability to successfully attend to the whole set of input -the simultaneous use of spoken (audio/soundtrack) and written (interlingual and intralingual subtitles) input modes -could depend, to some degree, on the language in them, especially if that language were to be one's L1, which in turn would require less effort in terms of processing and reading efficiency.…”