When words and graphs move the eyes:The processing of multimodal causal relations

Abstract

Research on causal relations in multisemiotic texts constituted by words and graphs has been scarce with only a few exceptions. In the current study, eye movement behavior was studied in seventy-six Chilean high school students, who read a set of twelve causally-related economics texts in Spanish in four experimental conditions. The objective is twofold. We aimed, on the one hand, to observe the main effects of the causal discourse marker (DM) por tanto and the statistical causal graph (G), as well as the interaction effect of both variables on different eye tracking measures. On the other, we seek to observe the effects of the DM on the same eye tracking measures for the graph system (GS) area of interest (AOI). The findings showed that the conjoint presence of the DM and the G did not positively influence the processing of selected AOIs. Analyses also reveal no significant effects on the GS AOI. Thus, the results indicate that the DM tend to decrease processing times, while the G increases them. Additional analyses conducted on the integrative transitions between the verbal system and the graph system reveal that more transitions were identified between the consequence segment and the graph system, thereby confirming that the consequence segment is crucial for the integration of both semiotic systems.

Always look back: Eye movements as a reflection of anaphoric encapsulation in Spanish while reading the neuter pronoun ello

“…Besides, the fact that the processing of the pronoun ello did not show differences between the two conditions could be connected to other experiments with similar results, where different Spanish connectives (por tanto and por eso) were studied (Recio et al, 2018). Since these connecting units have fundamentally procedural meaning, they require other elements with conceptual meaning through which they can fulfill their instruction.…”

“…For decades, research into the psycholinguistics of pronoun interpretation has argued that readers employ diverse interpretation preferences or routes associated with the specific linguistic properties of the antecedents, particularly focusing on third person gender-specific pronouns and on distance variables (i.e., Ehrlich, 1980;Duffy and Rayner, 1990;Kehler et al, 2008;Stewart et al, 2007). However, there is limited literature available focusing on the processing of neuter pronouns; to our knowledge, the only exceptions are on Spanish (Loureda et al, 2015;Recio et al, 2018), and on English (Brown-Schmidt et al, 2005;Schuster, 1988).…”

Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows

“…A relevant term is 'representational modality,' frequently used for text, figures, and diagrams of various types (Acartürk, 2014;Bernsen, 1994;Bateman, 2008, cf. 'multimodal documents';Parodi, Julio & Recio, 2018). Accordingly, the research on multimodal communication emphasizes one aspect of multimodality or the other, depending on the research context and the specific target of the study.…”

“…The instructional material usually involves multiple representations, such as text and figures aiming to transform the material into comprehensible chunks in order to facilitate learning, depending on the level of learner (cf. 'intermodal connections' in Parodi, et al, 2018). Nevertheless, as Parodi (2012) states, the research on specific types of multimodal interaction, such as a systematic investigation of multisemiotic descriptions of written texts and diagrams, is scarce.…”