Multimodal Comprehension of Language and Graphics: Graphs with and without annotations

Abstract

An experimental investigation into interaction between language and information graphics in multimodal documents served as the basis for this study. More specifically, our purpose was to investigate the role of linguistic annotations in graph-text documents. Participants were presented with three newspaper articles in the following conditions: one text-only, one text plus non-annotated graph, and one text plus annotated graph. Results of the experiment showed that, on one hand, annotations play a bridging role for integration of information contributed by different representational modalities. On the other hand, linguistic annotations have negative effects on recall, possibly due to attention divided by the different parts of a document.

How the poor get richer: Signaling guides attention and fosters learning from text‐graph combinations for students with low, but not high prior knowledge

“…We found no evidence for the potential alternative assumption that multimedia integration signals may be ineffective for learning with text-graph material due to its potential disruptive function regarding graph interpretation processes (cf. Acartürk et al, 2008). (Haider & Frensch, 1999).…”

“…Rather, they usually require verbal information such as axis labels and explanatory accompanying text in order to be comprehensible (cf. Acartürk et al, 2008). Accordingly, in CTML terms, graphs place high demands onto organizing the graphical information into a mental model, which is otherwise just seen as an albeit necessary prerequisite for the pivotal process of text-picture integration.…”

“…In particular, signals may guide attention before the graph's syntax has been fully understood, thereby disrupting necessary graphinterpretation processes. The more intense processing of the graph when no signals were present as observed by Acartürk et al (2008) suggest that students focused on the graph to understand its syntax and only then contextualized it with the help of the text. On the other hand, shifting attention between text and graph before its syntax has been fully understood may interfere with graph comprehension.…”

When words and graphs move the eyes:The processing of multimodal causal relations

“…Additional analyses conducted on the integrative transitions between the verbal system and the graph system reveal that more transitions were identified between the consequence segment and the graph system, thereby confirming that the consequence segment is crucial for the integration of both semiotic systems. Domínguez, 2007;Briz, Pons, & Portolés, 2017;Loureda & Acín, 2010), but also through psycholinguistic cognitive processing research (Sanders, Spooren, & Noordman, 1992, 1993van den Broek, 1990;Graesser & Bertus, 1998;Otero, León, & Graesser, 2002;Acartürk, Habel, Cagitay, & Alacam, 2007;Parodi, 2005;Canestrelli, Mak, & Sanders, 2013). Nevertheless, research on causal relations in multisemiotic texts constituted by words and graphs is scarce with only a few exceptions (Green, 2005;Habel & Acartürk, 2011).…”

“…Furthermore, the visual representation of data facilitates the interpretation and comprehension of numerical or statistical values associated with variables under study. The subsequent wide dissemination and frequent use of graphs have motivated researchers to identify and describe the different cognitive processes involved in the reading of graphs (Acartürk et al, 2007;Hegarty, Canham, & Fabrikant, 2010;Schnotz & Baadte, 2015;Strobel, Sass, Lindner, & Koller, 2016), and to develop theoretical-empirical models (e.g., Cleveland & McGill, 1984;Pinker, 1982Pinker, , 1990Winn, 1994;Shah, 1997;Mayer, 2005Mayer, , 2010.…”