Multimodal digital semiotics: the interaction of language with other resources

Abstract

A “multimodal digital semiotics” approach, involving the development and use of interactive digital media technology, mathematical techniques of analysis and scientific visualization for modeling and analyzing multimodal phenomena, is demonstrated through the analysis of an interview between a climate scientist and a climate denialist on Fox News. Prototype software for multimodal analysis and network visualizations are used in combination with Allen's interval algebra to model multimodal semantic data and identify significant patterns based on temporal relationships between different semiotic choices. The resulting semantic patterns reveal combinations of semiotic choices that were drawn upon by the different speakers during the interview, accounting for how the climate denialist emerges as the dominant and convincing interviewee. Such an approach combines the objectivity of mathematical modeling and scientific visualization with the interpretive power of Halliday's systemic theory.

A Digital Mixed Methods Research Design: Integrating Multimodal Analysis With Data Mining and Information Visualization for Big Data Analytics

“…The resulting multimodal data, stored and retrieved from such platforms, can then be modeled and further analyzed using a range of mathematical and computational techniques, as found in other mixed methods designs. For example, singular value decomposition, k -means clustering and temporal logic have been used to detect semantic patterns in multimodal texts which would not otherwise be discernible; for example, O’Halloran, E, Podlasov, and Tan (2013) and O’Halloran, E, and Tan (2014) characterize in detail the differences in the multimodal strategies deployed by a climate scientist and a climate denialist and how they are visually portrayed in a televised debate about climate change, illustrating how the approach can be used for other studies.…”

Multimodal literacy: Unfolding reading path in children’s picture book and its potential for EFL classrooms

“…The reading path is defined as the order and direction in which the reader approaches the text or images on the page (O'Halloran, Marissa, Podlasov & Tan, 2013). Then, the prescribed reading path is divided into two categories, namely linear and non-linear reading paths.…”

Editorial: Introduction to this Special Issue: English for Academic and Professional Purposes in the Digital Era

“…LSP research has, of course, long employed computerized corpus tools to quantitatively examine the features of specialized communication. More recent advances now allow researchers to analyse multimodal corpora (O’Halloran et al, 2013), analyse writing processes using tools like keylogging and screen recording software (Perrin, 2016), and analyse user behaviour in online communities using tools of big data to collect evidence and quantitative network analysis to understand it (Page, 2016). Some of these innovations are still waiting to be fully exploited in LSP research.…”