A Cognitive Model and a Knowledge Representation System for Music and Multimedia

Abstract

This article introduces a cognitive model and an experimental system for the representation and real-time processing of music and multimedia, based on both artificial intelligence (AI) and traditional techniques. First, it introduces some basic issues on the requirements of Al-based music systems and the class of problems faced by our system (music and multimedia composition, real-time performance, and analysis). Then, our proposal of a representation scheme -on the basis of an implemented system, called HARP -is described. It is based on the integration of different formalisms, able to manage the different nature and levels of music and multimedia objects, from the symbolic, knowledge level, dealing with multilevel, abstract representations and reasoning mechanisms, to the subsymbolic level, dealing with low-level processes, images, and physical signals. In our model, the symbolic level integrates a semantic network language of the family of KL-ONE with a temporal logic language and production rules. The subsymbolic level is modelled as a set of agents, mainly based on dynamic systems: some particular examples on the use of abstract potentials are described here. According to our bottom-up approach to AI and multimedia, we are mainly interested in the study and development of "autonomous" multimedia systems, that is, autonomous agents characterized by multimodal interaction with user(s) in unstructured and evolving environments. These particular scenarios include a theatrical automation project, where the system is delegated to manage and integrate sound, music and three-dimensional computer animation of humanoid figures interacting with real actors on stage, and a museal application, in which the system controls in real time the behaviour of a sensorbased autonomous mobile system in an exposition area, able to welcome, entertain, guide and instruct visitors.

Expressiveness and Physicality in Interaction*

“…The mapping strategies of expressive multimodal input onto multimedia output, for example by adopting shared internal space-time representations of perceptual data from different channels (acoustic, visual, gestural), is of paramount importance. Common space-time representations (Camurri & Leman, 1997, or the Laban's Effort Theory computational model described in Camurri, Trocca & Volpe, 1999), abstractions on multimodal perceptual data, computational models based on metaphors (e.g., the potential fields described in Camurri et al, 1994) and Gestalt-based approaches (Leman, 1995) are in our view particularly significant in this multimodal perspective.…”

“…In (Camurri et al, 1994) we defined a cognitive model and a software environment for composition and performance (HARP) based on agents and metaphors of energy fields. The use of metaphors as a``glue'' between modalities, i.e., a means unifying in a multimodal perspective the different languages involved (music, gesture, visual) was also one of the main motivation issues in HARP and now in EyesWeb.…”

Interactive environments for music and multimedia

“…The audio compact disc from IEEE CS that accompanied the July 1991 issue of the IEEE Computer magazine includes several music examples produced with an earlier version of the system designed for computer-aided composition (Camurri et al 1991). A previous version of the system is described in (Camurri et al 1994(Camurri et al , 1995a.…”

“…As a part of a general investigation on multimodal interaction in multimedia systems (the Esprit Basic Research Project 8579 MIAMI -Multimodal Interaction for Advanced Multimedia Interfaces), our research is mainly concerned with the study and development of ME architectures. This modeling effort is synergic with a stream of software developments, previously grouped in the HARP (Hybrid Action Representation and Planning) project (Camurri et al 1991(Camurri et al , 1994(Camurri et al , 1995 and recently evolved to the current architecture. Some of our recent results, both in the research and in practical effective system applications, are reviewed in this paper.…”

Timing in music and modal temporal logic

“…This kind of representation is crucial in the modelling of listening processes, and in the design of automata to perform or create music in real-time set-ups. Relevant previous work exists in the theoretical formulations of Kunst [2][3] and Leman [4][5][6], with respect to the modelling of the listening experience, and in real-time software systems with a strong theoretical base such as HARP [7][8] and various systems based on Petri nets [9][10]. The objective of this paper is to enhance the mathematical rigour underlying such work by formulating concepts based on temporal logic which might provide for musical systems the kind of basis for proof and formal design which it provides for computer science (see [11]).…”