A multimodal approach to visual thinking: the scientific sketchnote

Abstract

There is a growing interest in the use of visual thinking techniques for promoting conceptual thinking in problem solving tasks as well as for reducing the complexity of ideas expressed in scientific and technical formats. The products of visual thinking, such as sketchnotes, graphics and diagrams, consist of ‘multimodal complexes’ that combine language, images, mathematical symbolism and various other semiotic resources. This article adopts a social semiotic perspective, more specifically a Systemic Functional Multimodal Discourse Analysis approach, to study the underlying semiotic mechanisms through which visual thinking makes complex scientific content accessible. To illustrate the approach, the authors analyse the roles of language, images, and mathematical graphs and symbolism in four sketchnotes based on scientific literature in physics. The analysis reveals that through the process of resemiotization, where meanings are transformed from one semiotic system to another, the abstractness of specialized discourses such as physics and mathematics is reduced by multimodal strategies which include reformulating the content in terms of entities which participate in observable (i.e. tangible) processes and enhancing the reader/viewer’s engagement with the text. Moreover, the compositional arrangement creates clear stages in the development of the ideas and arguments that are presented. In this regard, visual thinking is a form of cultural communication through which abstract ideas are translated and explained using a multimodal outline or summary of essential parts by adapting resources (e.g. linguistic resources and mathematical graphs), using new resources (e.g. stick figures and other simple schematic drawings) and maintaining others from the original text (e.g. mathematical symbolic notation), resulting in a congruent (or concrete) depiction of abstract concepts and ideas for a non-specialist audience.

An Eye-Tracking Study of Sketch Processing: Evidence From Russian

“…Paivio (2006) indicates that the information is represented in the memory by a text and a corresponding illustration, not just by a text. It is assumed that the information in a polycode text is doubledecoded: the concept of an image is "superimposed" on the concept of a verbal text, the interaction of these two concepts leads to the creation of a general concept (meaning) of the text (Telminov, 2009;Fernández-Fontecha et al, 2018). Independent parts of a polycode text interact and create a "holistic experience, " the combination of the visual language with the written language.…”

The Challenges and Opportunities of Energy-Flexible Factories: A Holistic Case Study of the Model Region Augsburg in Germany

“…A sketchnote is a kind of mind map with illustrations [18]. To create a common ground for the one-day stakeholder dialogue, a Sketchnote including a glossary of important terms was created with the following key questions: What is the energy transition?…”

Thinking Differently: A Visual Note Recording Strategy to Improve Learning

“…It is important component of today's educational environment because of the tremendous expansion in the use of smart technologies that employ the Internet, smart phones, and computers (Tillmann, 2012;Matusiak, Heinbach, Harper & Bovee, 2019). Visual note recording helps improve visual literacy skills as well as visual thinking (Walker, 2012;Tillmann, 2012;Fernández-Fontecha et al 2016). The ability to analyze and interpret visuals alone "is not by itself sufficient for full visual literacy; it must be accompanied by some ability to create visual material" (Brumberger, 2011, p. 21).…”

“…VNR then becomes another strategy to help us think, comprehend, and recall information (Beatty, 2013;Brown, 2014;Mead 2019). In addition, some research adds that visual note recording during a learning setting, if used purposefully, would stop the person from day dreaming (Andrade, 2010;Fernández-Fontecha et al, 2016). On the other hand, Chan (2012) shows no significant effect on his learner regarding using VNR spontaneous drawing and the subsequent recall of information.…”

“…There is the need to redefine the concept as a purposeful activity that can improve the learning process. It is also important to consider VNR as a communication learning tool, and a multimodal approach to visual thinking that can help prompt memory and develop a higher level of understanding (Zucker, 2019;Fernández-Fontecha, O'Halloran, Tan & Wignell, 2016). Moreover, VNR involves visual listening and visual idea generation .…”