Immersive virtual reality (VR) for digital media making: transmediation is key

Mills, Kathy A.; Brown, Alinta

doi:10.1080/17439884.2021.1952428

Cited by 31 publications

(24 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The interaction and interplay of semiotic modes, often described as intersemiosis (Royce, 1998;O'Halloran 2008), result in a multiplication of meanings (Lemke 1998). The intermodal relations, such as image-text relations (Liu and O'Halloran 2009;Unsworth 2001;2013), and languagegesture relations (Martinec 2004;Hood 2011;Lim 2021b), and transmediation (Mills 2011;Mills and Brown 2021) have been the topic of study for many scholars working in multimodal studies. Of particular interest is the concept of "transduction" (Kress 2010) and "transmodal transformation" (Newfield 2014) which have been used in multimodal studies to examine shifts or resemiotization (Iedema 2003) in meanings across different modes.…”

Section: A Pedagogic Metalanguage Of Transpositionmentioning

confidence: 99%

A Metalanguage for Learning: Rebalancing the Cognitive with the Socio-Material

2022

View full text Add to dashboard Cite

Technology has enabled new ways of meaning-making in the digital age, incidentally bringing with it inequities in education as a result of the differing access, resources, and experiences of students. These inequities may be rendered invisible if society and schools neither recognize, value nor set out to include in formal education the meaning-making practices from students’ lifeworlds. Such neglect can perpetuate the digital divide among students from diverse home backgrounds. The reform agenda of multiliteracies is to bring about educational justice through a pedagogy of access. In this paper, we discuss how this agenda can be operationalized in the frontline of education—the classroom. We propose a pedagogic metalanguage of transpositional grammar for the learning of multimodal literacy. “Transposition” refers to the process of moving between different forms of meaning (text, image, space, object, body, sound and speech), and changes of attention to their functions (reference, agency, structure, context and interest). In particular, we show the value of having a common shared conceptual framework with which to reflect upon and unpack multimodal meaning in terms of its forms and functions. We also describe how a repertoire of knowledge processes, rebalancing the cognitive and the socio-material, affective and embodied, can support teachers in their design for students’ multimodal literacy learning. We argue that attention to multimodal literacy in the curriculum, pedagogy, and assessment can be productively supported with a pedagogic metalanguage of transpositional grammar and discuss how this can be a step towards mediating the invisible inequities in education in the digital age.

show abstract

Section: A Pedagogic Metalanguage Of Transpositionmentioning

confidence: 99%

A Metalanguage for Learning: Rebalancing the Cognitive with the Socio-Material

2022

View full text Add to dashboard Cite

show abstract

“…It was interesting to note that while there was unanimous consensus regarding the positive influence of I-VR on learning and understanding, the students were unclear as to how I-VR “tricked” the brain into creating the illusion of spatial parity (Mills and Brown 2021 ). In essence, the quality of the simulation was sufficient to not cause students to “reject” the illusion and potentially not engage meaningfully.…”

Section: Resultsmentioning

confidence: 99%

Developing student codesigned immersive virtual reality simulations for teaching of challenging concepts in molecular and cellular biology

Reen

Jump

McEvoy

et al. 2022

FEMS Microbiology Letters

View full text Add to dashboard Cite

Molecular biology theory represents a critical scaffold which underpins multiple disciplines within life sciences education. However, it is well documented that undergraduate students can struggle to achieve deeper understanding of key concepts and/or their application. One challenging, contributory aspect is the “invisible” nature of molecular biology processes compounded by critical three-dimensional spatial orientations of the principal components and their interactions. Molecular theory specifically requires students to construct accurate, mental spatial models to develop their understanding. However much of the traditional teaching and examination of such theory is limited to two-dimensional representations. Technology-enhanced, complementary teaching and examination approaches, which engage students with spatial aspects of theoretical concepts, offer an exciting opportunity to support student learning in this area. In this study we have explored the integration of an immersive virtual reality simulation based on a challenging molecular biology concept within an existing module taught at University College Cork. A mixed methods approach, grounded in learning theory, was undertaken to assess the student user and learning experience. The consensus response from students was one of enhanced learning, understanding, engagement, and motivation. Student partnership in the process of simulation design and integration was key to delivering the fully integrated experience.

show abstract

“…VR technologies offer different affordances for embodied writing and multimodal communication, distinguished from previous forms of virtual technology with the advent of HMDs that have become prominent since 2013 (Jensen & Konradsen, 2018). These affordances account for multimodal and sensorial ways of interacting with sophisticated, three-dimensional virtual environments and objects, orchestrating vision, sound, haptics (touch to interact with the environment), and head and body movements, in fully immersive environments—which differs from being seated at a computer screen as an observer of one’s avatar in a virtual world (Mills & Brown, 2021; Minogue & Jones, 2006).…”

Section: Multimodal Communication and Vr Representationmentioning

confidence: 99%

“…Expanding repertoires of digital media practices are now required by users of all ages, for youth and beyond (Sefton-Green & Erstad, 2017), which now includes meaning making in virtual reality (VR) contexts. VR technologies offer new potential for users to engage in immersive multimodal communication practices within and across virtual and material spaces, which remain underresearched (Henriksen et al, 2021) and which have unexplored implications for communication using novel forms of sensory-motor engagement (Mills & Brown, 2021). Multimodal refers to the combination of two or more semiotic modes or cultural means of representation, such as images, sounds, words, or body movements (Mills et al, 2018), and it is in the relationships between co-present modes that the “expressive power of multimodality resides” (Hull & Nelson, 2005).…”

Section: Introductionmentioning

confidence: 99%

Virtual Reality and Embodiment in Multimodal Meaning Making

Mills

Scholes

Brown

2022

Written Communication

Self Cite

View full text Add to dashboard Cite

Immersive virtual reality (VR) technology is becoming widespread in education, yet research of VR technologies for students’ multimodal communication is an emerging area of research in writing and literacies scholarship. Likewise, the significance of new ways of embodied meaning making in VR environments is undertheorized—a gap that requires attention given the potential for broadened use of the sensorium in multimodal language and literacy learning. This classroom research investigated multimodal composition using the virtual paint program Google Tilt Brush™ with 47 elementary school students (ages 10–11 years) using a head-mounted display and motion sensors. Multimodal analysis of video, screen capture, and think-aloud data attended to sensory-motor affordances and constraints for embodiment. Modal constraints were the immateriality of the virtual text, virtual disembodiment, and somatosensory mismatch between the virtual and physical worlds. Potentials for new forms of embodied multimodal representation in VR involved extensive bodily, haptic, and locomotive movement. The findings are significant given that research of embodied cognition points to sensorimotor action as the basis for language and communication.

show abstract

Immersive virtual reality (VR) for digital media making: transmediation is key

Cited by 31 publications

References 30 publications

A Metalanguage for Learning: Rebalancing the Cognitive with the Socio-Material

A Metalanguage for Learning: Rebalancing the Cognitive with the Socio-Material

Developing student codesigned immersive virtual reality simulations for teaching of challenging concepts in molecular and cellular biology

Virtual Reality and Embodiment in Multimodal Meaning Making

Contact Info

Product

Resources

About