Locating the Semiotic Power of Multimodality

Abstract

This article reports research that attempts to characterize what is powerful about digital multimodal texts. Building from recent theoretical work on understanding the workings and implications of multimodal communication, the authors call for a continuing empirical investigation into the roles that digital multimodal texts play in real-world contexts, and they offer one example of how such investigations might be approached. Drawing on data from the practice of multimedia digital storytelling, specifically a piece titled “Lyfe-N-Rhyme,” created by Oakland, California, artist Randy Young (accessible at http://www.oaklanddusty.org/videos.php), the authors detail the method and results of a fine-grained multimodal analysis, revealing semiotic relationships between and among different, copresent modes. It is in these relationships, the authors argue, that the expressive power of multimodality resides.

An Empirical Study of Ecuadorian University EFL Learners’ Comprehension of English Idioms Using a Multimodal Teaching Approach

“…For the next exercise, students were asked to work with a partner to produce an artistic interpretation of one of the idioms selected at random by means of a digital collage. Having students compose in a digital, multimodal landscape can present "not just a new way to make meaning, but a different kind of meaning" (Hull and Nelson, 2005, p. 225). Students had to guess which idiom was being represented by each digital collage.…”

La production vidéo : une pratique multimodale pour tisser des liens entre l’école et les littératies hors scolaires

“…Selon eux, l'école doit prendre en compte les pratiques de littératie adoptées à l'extérieur de l'institution scolaire afin de mieux préparer les enfants à vivre dans un monde en pleine mutation marqué par la diversité linguistique et culturelle, la globalisation des marchés ainsi que par l'expansion rapide des (Ajaji, 2009;Brass, 2008;Bruce, 2009;Carrington et Robinson, 2009;Hoechsmann et Poyntz, 2012;Hull et Nelson, 2005;Kalantzis et Cope, 2013;Kendrick, McKay et Moffat, 2005;Kress, 2003;Lebrun, Lacelle, Boutin, 2012;Pahl et Rowsell, 2006;Stein, 2004).…”

The Arts and Education: Knowledge Generation, Pedagogy, and the Discourse of Learning

“…At one and the same time, there is movement toward expansive and multilayered ways of thinking, centered in cultural and social practices and diversity of visual texts and art forms, and there is movement away from codifying the arts into discrete categories (Alim & Baugh, 2007;Dartnall, 2002;Flood, Heath, & Lapp, 2005;Greene, 2000;Hull & Nelson, 2005). These new references highlight the organic nature of the arts, challenge simplistic notions of product and process, and promote a view in which the varied substance and enactments of the arts are studied and understood in relationship to where and how they are situated in the human experience and in individuals' experiences as members of cultural and social collectives (Begoray, 2001).…”

“…Hull and Nelson (2005) present a core of compelling arguments in an article on multimodality-speech, writing, image, gesture, and sound. Although it is not described in terms of the arts, I include it here because of its focus on literacy and performance and because of the complexity of the work that emerged from youth in an out-of-school program as they created a kind of textual art.…”

“…Through this work, Hull and Nelson (2005) offer a perspective on methodologytheir own described as "roughly within the tradition of design experiments, whereby program development is intertwined with continual attempts to assess and improve . .…”