Sensing and mapping for interactive performance

Abstract

This paper describes a trans-domain mapping (TDM) framework for translating meaningful activities from one creative domain onto another. The multi-disciplinary framework is designed to facilitate an intuitive and non-intrusive interactive multimedia performance interface that offers the users or performers real-time control of multimedia events using their physical movements. It is intended to be a highly dynamic real-time performance tool, sensing and tracking activities and changes, in order to provide interactive multimedia performances.From a straightforward definition of the TDM framework, this paper reports several implementations and multi-disciplinary collaborative projects using the proposed framework, including a motion and colour-sensitive system, a sensor-based system for triggering musical events, and a distributed multimedia server for audio mapping of a real-time face tracker, and discusses different aspects of mapping strategies in their context. Plausible future directions, developments and exploration with the proposed framework, including stage augmentation, virtual and augmented reality, which involve sensing and mapping of physical and non-physical changes onto multimedia control events, are discussed.

IStage

“…There have been several attempts to build systems for controlling graphics and sounds using user gestures or emotions [7,8,10]. The closest to our system is Cyclic, a dancer-controlled augmentation of the performance through graphics and music, using the dancer movements as input data to these components [1].…”

LIVeMotion: A Multi-Sensory System to Encourage the Awareness of Mindfulness

“…The system can be configured to make use of different mapping strategies and appropriate feedback for sensor control of an artistic performance, similar to the concept of trans-domain mappings for "Music via Motion" (Ng 2002, Ng 2004, Leonard, James & Ng 2011, Ng 2011). …”

Interface Devices and Software Tools for Expressive Music Creation

“…Especially, a video-to-music system has been recognition, all the characteristic parameters are extracted emphasized with the aim of investigating how to associate and the normalized distance is calculated for each standard natural gestures with sound/musical features for interactive pattern data [22]. performance, for example [9] [15] [16] [17]. In the past some…”