The SemanticHIFI project: content-based management and manipulation of musical recordings

Abstract

The SemanticHIFI project aims at designing and prototyping tomorrow's Hi-fi systems, which will provide music lovers with innovative functions of management and manipulation of musical contents. The limitations of current equipments are mainly related to those of the music distribution media (album-based audio recordings in stereo format), with poor control features and interfaces (album/ track selection, play, stop, volume, etc.). Enabling the manipulation of richer media and related metadata (either distributed with the audio recordings or computed by the user using dedicated indexing tools) opens a wide range of new functionalities: personal indexing and classification of music titles, content-based browsing in personal catalogues, browsing within titles with automatic segmentation and de-mixing tools, 3D audio rendering and assisted mixing features, etc. Moreover, the manipulation of interactive music contents will be made accessible to music consumers, through dedicated performing, and authoring tools. They will then have the possibility of publishing and sharing their personal work with others, through a dedicated peer-to-peer sharing middleware specifically designed for preserving the rights of the used digital media. All these features are the result of the various R&D tasks and experiments performed as part of the project and represent the state-of-the art in various research fields : digital audio signal processing, music information retrieval, man-machine interfaces and peer-to-peer networks. Moreover, the project includes an integration phase, which aims at producing full-featured applications prototypes, designed to fit identified market needs, through technical choices compatible with these markets. This article proposes an overview of the project, by presenting its background and objectives, the main scientific issues and breakthroughs it addresses in relation to the description and extraction of musical information, the main applications features it aims at developing and choices made for their integration into application prototypes compliant with market needs.

Science and Technology of Music and Sound: The IRCAM Roadmap

“…IRCAM was involved in the coordination of the first large-scale music indexing R&D projects carried out at a French national level with the ECRINS project [Mullon et al 2002] and at a European level with the CUIDADO project (2001)(2002)(2003) that systematically investigated all possible applications of the automatic extraction of musical contents from signal and symbolic materials [Vinet et al 2002] and proposed inputs to the MPEG7 standard [ISO/IEC 2002, Peeters et al 2000. More recently, the SemanticHIFI European project 9 was dedicated to designing future Hi-Fi systems through the experimentation of advanced feature-based interfaces for the manipulation of musical recordings [Vinet 2005[Vinet , 2006. Numerous advances resulted from these studies, concerning the automatic extraction of various features, systems for the automatic classification of sounds, and the automatic extraction of the temporal structure of songs [Peeters 2005[Peeters , 2006a[Peeters , 2006b[Peeters , 2007a[Peeters , 2007b.…”

“…The current reconfiguration of the music distribution system employing digital networks makes it possible, for the first time, to overcome a number of limitations of the exchange formats toward rich data sets that could include multi-channel audio data, symbolic data, and metadata and whose transmission could be scaled down to match the capacities of client terminals. The availability of rich musical data on client terminals could yield new interactive interfaces for browsing within the musical material as demonstrated in the SemanticHIFI project [Vinet 2006] or for collaborative distributed performance or production. These interfaces may feature a form of fusion of the functions of listening devices and electronic instruments that have traditionally been separated in the consumer electronic market.…”

Rhythm metadata enabled intra-track navigation and content modification in a music player

“…Using music content descriptive metadata for facilitating content manipulation for amateur listeners is an emerging research area. For an overview of some earlier approaches see Vinet [2005].…”