Measuring embodied conceptualizations of pitch in singing performances: Insights from an OpenPose study

Abstract

People conceptualize auditory pitch as vertical space: low and high pitch correspond to low and high space, respectively. The strength of this cross-modal correspondence, however, seems to vary across different cultural contexts and a debate on the different factors underlying this variation is currently taking place. According to one hypothesis, pitch mappings are semantically mediated. For instance, the use of conventional metaphors such as “falling” or “rising” melodies strengthens a pitch-height mapping to the detriment of other possible mappings (e.g., pitch as bright/dark color or small/big size). Hence, entrenched pitch terms shape specific conceptualizations. The deterministic role of language is called into question by the hypothesis that different pitch mappings share a less constraining conceptual basis. As such, conceptual primitives may be concretized ad hoc into specific domains so that more local variation is possible. This claim is supported, for instance, by the finding that musicians use language-congruent (conventional) and language-incongruent (ad hoc) mappings interchangeably. The present paper substantiates this observation by investigating the head movements of musically trained and untrained speakers of Dutch in a melody reproduction task, as embodied instantiations of a vertical conceptualization of pitch. The OpenPose algorithm was used to track the movement trajectories in detail. The results show that untrained participants systematically made language-congruent movements, while trained participants showed more diverse behaviors, including language-incongruent movements. The difference between the two groups could not be attributed to the level of accuracy in the singing performances. In sum, this study argues for a joint consideration of more entrenched (e.g., linguistic metaphors) and more context-dependent (e.g., musical training and task) factors in accounting for variability in pitch representations.

The materiality key: how work on empirical data can improve analytical models and theoretical frameworks for multimodal discourse analysis

“…) makes the examples of empirical data analysis proposed here particularly appropriate to the consideration of the notion of materiality as an interdisciplinary one and provides a clear connection with John Bateman's discussion of materiality in relation to the development of Multimodality as a practice that encompasses borders between disciplines and research areas (Bateman et al, 2017;Bateman, 2019Bateman, , 2022. This article will also show how the consideration of the materiality of dance allowed in primis for the further development of the Functional Grammar of Dance (Maiorani, 2021;Maiorani and Liu, 2023), which is now a more comprehensive and even more flexible tool that scholars have started to use for analysing movement-based communication in dance performances other than ballet or even outside the domain of dance altogether (see Mouard Ruiz, 2021;Bolens, 2022;Meissl et al, 2022;Prové, 2022;Sindoni, 2022;Vidal Claramonte, 2022;Wu, 2022;Elyamany, 2023). The examples of empirical data analysis will be preceded by a presentation of how the Functional Grammar of Dance is implemented in ELAN, a widespread commercial, free-to-use software traditionally used for annotating conversations or verbal interactions, for which we created a complete set of interdependent tiers and controlled vocabulary.…”

Landscapes of coarticulation: The co-structuring of gesture-vocal dynamics in Karnatak music performance

“…However, as with co-speech gesturing [4,6], certain gestural forms and tendencies are picked up through enculturation, and so some similarities can be seen across performers [3]. There is a growing body of work on gesture in Indian music contexts, largely focussing on the North Indian, Hindustani style [3,[7][8][9][10][11][12][13], in addition to existing research on gesture in Western art music and choral contexts [14][15][16][17].…”