MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora

Abstract

This paper presents a collection of tools (and adaptors for existing tools) that we have recently developed, which support acquisition, annotation and analysis of multimodal corpora. For acquisition, an extensible architecture is offered that integrates various sensors, based on existing connectors (e.g. for motion capturing via VICON, or ART) and on connectors we contribute (for motion tracking via Microsoft Kinect as well as eye tracking via Seeingmachines FaceLAB 5). The architecture provides live visualisation of the multimodal data in a unified virtual reality (VR) view (using Fraunhofer Instant Reality) for control during recordings, and enables recording of synchronised streams. For annotation, we provide a connection between the annotation tool ELAN (MPI Nijmegen) and the VR visualisation. For analysis, we provide routines in the programming language Python that read in and manipulate (aggregate, transform, plot, analyse) the sensor data, as well as text annotation formats (Praat TextGrids). Use of this toolset in multimodal studies proved to be efficient and effective, as we discuss. We make the collection available as open source for use by other researchers.

Head movement and its relation to hearing

“…Head movements in communication settings comprise a variety of head movement patterns with a correspondingly wide range of culturally specific meaning and interpretation. For most western cultures, the most common communicative head movements are “nods” and “shaking,” both of which involve oscillations along the rotational axes ( Kousidis et al, 2013 ; Wagner et al, 2014 ; Hall et al, 2019 ), though the meaning of these movements can vary by culture ( Andonova and Taylor, 2012 ; Wagner et al, 2014 ).…”

“…Head shaking has a similar potential for increasing spatial information and serves as a means of non-verbal communication. The nuances that carry specific meaning in head movements can be difficult to study due to challenges in systematically defining and annotating specific behaviors across multiple observers ( Kousidis et al, 2013 ). Available evidence indicates that varying levels of intensity, velocity, and range are the key features that carry communication intent, transmitting signals such as feedback from a listener to a talker, indicating an interest in contributing to a conversation (turn-taking), and providing emphasis or subtlety to speech delivery ( Hadar et al, 1983 ; Wagner et al, 2014 ).…”

“…Head tilts also are noncyclical and differ from nodding, shaking, and turning in that the rotations are on the roll axis ( Kousidis et al, 2013 ; Wagner et al, 2014 ). In the context of non-verbal communication during conversation, the meaning of head tilts is harder to define and annotators often confuse them with turns ( Kousidis et al, 2013 ), but have been proposed to indicate disbelief, uncertainty, and skepticism ( Stone and Oh, 2008 ; Wagner et al, 2014 ).…”

“…For most western cultures, the most common communicative head 10.3389/fpsyg. 2023.1183303 Frontiers in Psychology 07 frontiersin.org movements are "nods" and "shaking," both of which involve oscillations along the rotational axes (Kousidis et al, 2013;Wagner et al, 2014;Hall et al, 2019), though the meaning of these movements can vary by culture (Andonova and Taylor, 2012;Wagner et al, 2014). "Turn" and "tilt" head movements are also described with similar frequency of occurrence in communication environments, with each motion leading to a different pattern of movement around the pitch, yaw, and roll axes of rotations.…”

Placing Objects in Gesture Space: Toward Incremental Interpretation of Multimodal Spatial Descriptions

“…In the experiment, hand motion was tracked by a Leap sensor, a USB peripheral device composed of two monochromatic cameras and three LED infrared sensors. 2 The hand motion data was recorded with MINT Tools (Kousidis, Pfeiffer, and Schlangen 2013). Audio and video were recorded by a camera.…”