Aesthetic multimodality of speech and gesture: Towards its functional framework

Abstract

The study develops a functional multimodal approach to speech and gesture behavior to explore aestheticism in more and less staged discourse of cinema and interview. We hypothesize that cinema and interview employ the same communicative functions; however, these functions constitute different frameworks which contribute to the higher aesthetic potential of cinema. This approach allows to study the aesthetic via communicative functions frameworks in multimodal discourse. To establish the function frameworks in cinema and interview, we apply a contrastive functional analysis of speech and gesture in the highly ranked actors’ argumentative and descriptive monologues. With the help of variance and regression analysis, we explore the distribution of pragmatic and discourse-structuring functions (with sub-functions) in speech as contingent on pragmatic, deictic, representational and adaptive functions of gestures. The study confirms that cinematic discourse exploits fewer deictic, representational and adaptive gesture functions, whereas pragmatic gesture functions (especially emphatic ones) appear more frequently and are contingent on several pragmatic and discourse-structuring functions of argumentative and descriptive speech. Interview function frameworks display lower predictability, which shows higher spontaneity of gestures; however, there are specific gestures typical of interview (self-adaptors) which may serve as indicators of pragmatic functions of argumentation. The study also manifests individual variations within the function frameworks. Overall, cinema and interview display variance in replication and regularity of speech and gesture functions, which presumably helps create higher and lower aesthetic effects.

Staged and natural gesturing in argumentation and description

“…Additionally, several works suggest that gestures are manifested differently in female and male behavior. Overall, women tend to use more hand gestures [2,4,28] which are mostly functional rather than perceptual [13,18]. Men may use more fidgeting and express restless behavior in comparison to women [13].…”

“…Men may use more fidgeting and express restless behavior in comparison to women [13]. In a more recent work [18], it is stated that women spend more time on gesturing and use gestures with nouns, while men mostly accompany adjectives with gestures. The research reveals that the temporal alignment between hand and gesture in terms of what comes first, a word or a gesture, is the same for both genders: gestures precede the speech, however the time lags between hand movements and related speech are longer for men than women.…”