Interactionally Embedded Gestalt Principles of Multimodal Human Communication

Abstract

Natural human interaction requires us to produce and process many different signals, including speech, hand and head gestures, and facial expressions. These communicative signals, which occur in a variety of temporal relations with each other (e.g., parallel or temporally misaligned), must be rapidly processed as a coherent message by the receiver. In this contribution, we introduce the notion of interactionally embedded, affordance-driven gestalt perception as a framework that can explain how this rapid processing of multimodal signals is achieved as efficiently as it is. We discuss empirical evidence showing how basic principles of gestalt perception can explain some aspects of unimodal phenomena such as verbal language processing and visual scene perception but require additional features to explain multimodal human communication. We propose a framework in which high-level gestalt predictions are continuously updated by incoming sensory input, such as unfolding speech and visual signals. We outline the constituent processes that shape high-level gestalt perception and their role in perceiving relevance and prägnanz. Finally, we provide testable predictions that arise from this multimodal interactionally embedded gestalt-perception framework. This review and framework therefore provide a theoretically motivated account of how we may understand the highly complex, multimodal behaviors inherent in natural social interaction.

Conversational facial signals combine into compositional meanings that change the interpretation of speaker intentions

“…Specifically, the present study provides evidence that the interpretation of the individual signals does not directly predict the interpretation of the combinations of signals, which also supports the notion of non-additive, Gestalt-like interpretation. This non-additive contribution of multimodal signals is directly in line with a recent theoretical framework of face-to-face communication 19 , and thus provides, to the best our knowledge, the first evidence for such Gestalt-like multimodal utterance perception.…”

“…This is because while individual facial signals may combine into more complex messages in a way that they should be considered compositional, the psychological processes underlying their interpretation into meaningful messages may closely follow Gestalt-psychological processes, in particular the notion that “the whole is more than the sum of its parts”—where the original actually translated ‘not the same’ as the sum of its parts 39 . This fits well with recent theoretical framings of human multimodal language as Gestalt-like 15 , 19 , 40 – 43 . It also fits with the recent, more flexible notion of compositionality in the animal literature 28 , 30 .…”

“…hand gestures and body shifts). Also, the larger interactional embedding of an utterance, which includes discourse context, learned idiosyncrasies of the speaker, and the observer’s own affective state all likely shape utterance interpretation 19 . Finally, the present study only assessed how the general population responded to multimodal Gestalts.…”

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

“…For example, eyebrow frowns often accompany a raising voice pitch to signal the intention to ask a question ( Nota et al, 2021 ). Mechanisms of Gestalt perception ( Wagemans et al, 2012 ), social affordance ( Gallagher, 2020 ), and relevance ( Sperber and Wilson, 1995 ) may jointly contribute to the recognition of multimodal communicative gestalts ( Trujillo and Holler, 2023 ). Finally, the recognition of a specific social action may trigger top-down multilevel predictions about how the message will unfold in time.…”

“…Specifically, this supports a parallel processing framework whereby the beginning of the message simultaneously activates multiple potential interpretations (i.e., multimodal gestalts). As the message unfolds, concurrent bottom-up sensory processing and multilevel predictions iteratively refine each other toward a final gestalt solution ( Trujillo and Holler, 2023 ). Such a parallel account accommodates evidence that processing of communicative social actions starts early ( Redcay and Carlson, 2015 ), perhaps in parallel to semantic comprehension ( Tomasello et al, 2022 ).…”