Synchronization in Interpersonal Speech

Abstract

During both positive and negative dyadic exchanges, individuals will often unconsciously imitate their partner. A substantial amount of research has been made on this phenomenon, and such studies have shown that synchronization between communication partners can improve interpersonal relationships. Automatic computational approaches for recognizing synchrony are still in their infancy. In this study, we extend on previous work in which we applied a novel method utilizing hand-crafted low-level acoustic descriptors and autoencoders (AEs) to analyse synchrony in the speech domain. For this purpose, a database consisting of 394 in-the-wild speakers from six different cultures, is used. For each speaker in the dyadic exchange, two AEs are implemented. Post the training phase, the acoustic features for one of the speakers is tested using the AE trained on their dyadic partner. In this same way, we also explore the benefits that deep representations from audio may have, implementing the state-of-the-art Deep Spectrum toolkit. For all speakers at varied time-points during their interaction, the calculation of reconstruction error from the AE trained on their respective dyadic partner is made. The results obtained from this acoustic analysis are then compared with the linguistic experiments based on word counts and word embeddings generated by our word2vec approach. The results demonstrate that there is a degree of synchrony during all interactions. We also find that, this degree varies across the 6 cultures found in the investigated database. These findings are further substantiated through the use of 4,096 dimensional Deep Spectrum features.

Interactional synchrony: signals, mechanisms and benefits

“…Verbal in-person interactions presents a special case of multisensory integration in which temporal features are extracted from, for example, the sound of spoken words, lip movements and additional gestures of the head and hands ( Munhall et al , 2004 ). Although in this modern age, there are counter examples in which we rely on an isolated visual (email) or auditory (phone) channel, face-to-face communication remains a crucial element of our lives with privileged processing mechanisms that benefit from synchronization ( Amiriparian et al , 2019 ) via temporally coupled neural responses to frequency-specific features in the signals ( Jiang et al , 2012 ). Possibly such neuronal synchronization supports goal-relevant behavioral coordination, including the mimicry of grammatical forms ( Hasson et al , 2012 ).…”

Incorporating Interpersonal Synchronization Features for Automatic Emotion Recognition from Visual and Audio Data during Communication

“…Despite all these efforts, we believe that there still is room for improvement. We were motivated by psychological studies that indicated that humans also use interpersonal features to recognize others' emotions [16][17][18][19]. According to our best knowledge, although the previous automatic emotion recognition research paid great attention to the individual features, most studies did not pay attention to the interpersonal features, especially the time-lagged synchronization.…”

“…If a man frowns, it may mean sorrow. Studies also have shown that interpersonal features such as mutual gaze [16,17], body synchronization [18] and the synchronization of speech [19] will help us to recognize others' emotions. Here, the interpersonal features used in this study are defined as the interpersonal interaction activities (verbal or nonverbal) that happen consciously or unconsciously during communication.…”

Recognizing Perceived Interdependence in Face-to-Face Negotiations through Multimodal Analysis of Nonverbal Behavior

“…Finally, we plan to account for the interactive nature of conversations in modeling explicitly. A first step could be to extend our approach by incorporating features from both individuals for predictions, i.e., to address phenomena related to behavioral coordination, such as mimicry [2].…”

“…Technological systems can benefit from estimating human's conceptualization and experience of social situations as contextual information about an interaction to display intelligent behavior in such collaboration settings [55]. In particular, intelligent systems could use such information about social perceptions to meaningfully adapt their functionality and behavior by facilitating predictions of (1) what a human might do (2) why they might do it, as well as (3) how a user might experience a system's actions.…”