An Exchange Format for Multimodal Annotations

Abstract

This paper presents the results of a joint effort of a group of multimodality researchers and tool developers to improve the interoperability between several tools used for the annotation and analysis of multimodality. Each of the tools has specific strengths so that a variety of different tools, working on the same data, can be desirable for project work. However this usually requires tedious conversion between formats. We propose a common exchange format for multimodal annotation, based on the annotation graph (AG) formalism, which is supported by import and export routines in the respective tools. In the current version of this format the common denominator information can be reliably exchanged between the tools, and additional information can be stored in a standardized way.

AutoBAP: Automatic Coding of Body Action and Posture Units from Wearable Sensors

“…As it is an XML file and BAP is, in principle, compatible with other annotation tools, it should not be a problem to convert it to other formats. See Schmidt et al for a description of an effort to convert between annotation formats [26]. Other eye trackers could also be used with few adjustments as long as it provides a scene camera to track the interlocutor or some other means to extract relative orientation and position.…”

Search Strategies for Pattern Identification in Multimodal Data

“…Data Characteristics: The data of each participant were multi-channel events represented by time points and/or intervals, i.e., annotated multimodal data [18,29,32]. This is commonly referred to as multivariate symbolic interval series with a mixture of symbolic time sequence data [23].…”

Interactive relevance search and modeling

“…Data Characteristics: The data of each participant are multi-channel events represented by time points and/or intervals (or a mixture), i.e., multimodal data [22,32,36]. The events were annotated from their media sources either automatically and/or manually.…”