Interactive environments for music and multimedia

Abstract

Multimodal Environments (MEs) are systems capable of establishing creative, multimodal user interaction by exhibiting real-time adaptive behaviour. In a typical scenario, one or more users are immersed in an environment allowing them to communicate by means of full-body movement, singing or playing. Users get feedback from the environment in real time in terms of sound, music, visual media, and actuators, i.e. movement of semi-autonomous mobile systems including mobile scenography, on-stage robots behaving as actors or players, possibly equipped with music and multimedia output. MEs are therefore a sort of extension of augmented reality environments. From another viewpoint, an ME can be seen as a sort of prolongation of the human mind and senses. From an artificial intelligence perspective, an ME consists of a population of physical and as software agents capable of changing their reactions and their social interaction over time. For example, a gesture of the user(s) can mean different things in different situations, and can produce changes in the agents populating the ME. The paradigm adopted for movement recognition is that of a human observer of the dance, where the focus of attention changes according to the evolution of the dance itself and of the music produced. MEs are therefore agents able to observe the user, extract "gesture gestalts", and change their state, including artificial emotions, over time. MEs open new niches of application, many still to be discovered, including music, dance, theatre, interactive arts, entertainment, interactive exhibitions and museal installations, information atelier, edutainment, training, industrial applications and cognitive rehabilitation (e.g. for autism). The environment can be a theatre, a museum, a discotheque, a school classroom, a rehabilitation centre for patients with a variety of sensory/motor and cognitive impairments, etc. The ME concept generalizes the bio-feedback methods which already have found widespread applications. The paper introduces MEs, then a flexible ME architecture, with a special focus on the modeling of the emotional component of the agents forming an ME. Description of four applications we recently developed, currently used in several real testbeds, conclude the paper.

Social retrieval of music content in multi-user performance

“…As for music listening, Camurri [25] proposed an early pioneering system where the user full-body rhythmic movements were analysed in real-time and compared with the beat of a song (extracted from the MIDI music signal). In the game Ghost In The Cave [26], two groups of users compete to collaboratively communicate emotions by full-body movement and singing voice, in order to achieve the goals of the game.…”

Priming creativity through improvisation on an adaptive musical instrument

“…This presents a challenge, because playing an instrument typically necessitates years of practice. Building on previous work with adaptive instrument design [5,16,19], we present a novel digital instrument featuring an adaptive interface that facilitates music creation. Our adaptive instrument enables non-musicians and musicians alike to experience the act of music improvisation, which acts as a prime to improve performance on a subsequent creative task.…”

“…Exploring adaptive instruments and instruments that interact intelligently with the user is an established area of research [5,16,19]. Many developments in this area have enabled expert users to express musical ideas that were hitherto out of reach, but this does not necessarily assist individuals who are not music experts.…”

Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library

“…Main research directions include (i) multimodal analysis and classification of expressive gestures in musical signals and human movement, (ii) real-time generation and post-processing of audio and visual content depending on the output of the analysis, (iii) study of the interaction mechanisms and mapping strategies enabling the results of the (multimodal) analysis to be employed (transformed) in the process of automatic generation of audio and visual content, and possibly of behavior of mobile robots (e.g. a moving scenery on stage, a robot for museums) [2,3,4].…”