Remarks on Multimodality: Grammatical Interactions in the Parallel Architecture

Abstract

Language is typically embedded in multimodal communication, yet models of linguistic competence do not often incorporate this complexity. Meanwhile, speech, gesture, and/or pictures are each considered as indivisible components of multimodal messages. Here, we argue that multimodality should not be characterized by whole interacting behaviors, but by interactions of similar substructures which permeate across expressive behaviors. These structures comprise a unified architecture and align within Jackendoff's Parallel Architecture: a modality, meaning, and grammar. Because this tripartite architecture persists across modalities, interactions can manifest within each of these substructures. Interactions between modalities alone create correspondences in time (ex. speech with gesture) or space (ex. writing with pictures) of the sensory signals, while multimodal meaning-making balances how modalities carry “semantic weight” for the gist of the whole expression. Here we focus primarily on interactions between grammars, which contrast across two variables: symmetry, related to the complexity of the grammars, and allocation, related to the relative independence of interacting grammars. While independent allocations keep grammars separate, substitutive allocation inserts expressions from one grammar into those of another. We show that substitution operates in interactions between all three natural modalities (vocal, bodily, graphic), and also in unimodal contexts within and between languages, as in codeswitching. Altogether, we argue that unimodal and multimodal expressions arise as emergent interactive states from a unified cognitive architecture, heralding a reconsideration of the “language faculty” itself.

The study of emoji linguistic behaviour: an examination of the theses raised (and not raised) in the academic literature

“…As they are closed-class items (they cannot be modified), we should study their combinatorial properties. In their most recent paper, Cohn and Schilperoord (2022) answer affirmatively with emoji: they understand that linear sequences with only meaningful associations (as in the case of emojis) constitute simple grammars, visual linear grammars.…”

The Lexicon of Emoji? Conventionality Modulates Processing of Emoji

“…These items include not only semantic information but also correspondences to structural and phonological (or orthographic) information as well. This approach of the parallel architecture has subsequently been extended to lexical items in other modalities (Cohn, 2016; Cohn & Schilperoord, 2022); consistent with the original flexibility of the “lexical item,” these same long‐term memory representations can be established for items like gestural emblems (Ladewig, 2020; McNeill, 1992), traffic signs (Forceville, 2019), logos (Foroudi, Melewar, & Gupta, 2014), and emoji.…”

Visual languages and the problems with ideographies: A commentary on Morin

“…Many patterns of two-unit sequences persist to show causative before–after relations, contrasts, or analogies (Schilperoord & Cohn, 2022). Longer sequences often provide visual lists of related images, such as what is allowed in a park or on an airplane (Cohn & Schilperoord, 2022). Visual narratives also use recursive combinatorial structures for sequential images displaying structural features of linguistic grammars, but operating at a higher-level information structure than the organization of nouns and verbs (Cohn, 2013; Cohn & Schilperoord, 2022).…”

“…Longer sequences often provide visual lists of related images, such as what is allowed in a park or on an airplane (Cohn & Schilperoord, 2022). Visual narratives also use recursive combinatorial structures for sequential images displaying structural features of linguistic grammars, but operating at a higher-level information structure than the organization of nouns and verbs (Cohn, 2013; Cohn & Schilperoord, 2022). These visual narrative sequences are natural productions of sequential images, and the specific sequencing constructions they use have been shown to vary across cultures' comics (Cohn, 2019), again indicating culturally relative standardization, not universality.…”