A General Framework for Visualization of Sound Collections in Musical Interfaces

Abstract

While audio data play an increasingly central role in computer-based music production, interaction with large sound collections in most available music creation and production environments is very often still limited to scrolling long lists of file names. This paper describes a general framework for devising interactive applications based on the content-based visualization of sound collections. The proposed framework allows for a modular combination of different techniques for sound segmentation, analysis, and dimensionality reduction, using the reduced feature space for interactive applications. We analyze several prototypes presented in the literature and describe their limitations. We propose a more general framework that can be used flexibly to devise music creation interfaces. The proposed approach includes several novel contributions with respect to previously used pipelines, such as using unsupervised feature learning, content-based sound icons, and control of the output space layout. We present an implementation of the framework using the SuperCollider computer music language, and three example prototypes demonstrating its use for data-driven music interfaces. Our results demonstrate the potential of unsupervised machine learning and visualization for creative applications in computer music.

Quantum Anomaly Detection with a Spin Processor in Diamond

“…The original waveforms are firstly analyzed by the method of Mel-frequency cepstral coefficients (MFCCs), [38,39] and then processed by the feature extraction method to reduce the dimension of each sample while keeping most of the information. [40,41] Details of the pre-processing are shown in Supporting Information. [34] After pre-processing, each audio segment is represented by a two-dimension feature vector and is ready to be analyzed by quantum processor.…”

Agent-Based Music Live Coding: Sonic adventures in 2D

“…The value is then assumed to be a folder of audio samples. The samples are analyzed using the FluidCorpusMap library (Roma et al 2021). This library performs analysis and dimensionality reduction of the audio features and maps them to a grid using an assignment algorithm.…”

“…Corpus-based concatenative synthesis systems such as CataRT (Schwarz, Beller, Verbrugghe and Britton 2006) have traditionally been based on 2D scatterplots using scalar descriptors as axes. Several systems have explored dimensionality reduction and layout mapping for descriptor-based visualisation of sound collections in two dimensions (see Roma, Xambó, Green and Tremblay 2021, and references therein). These systems have generally been controlled through input devices such as mice, tablets, or other sensors.…”

“…These systems have generally been controlled through input devices such as mice, tablets, or other sensors. Some systems (Garber, Ciccola, and Amusategui 2020; Roma et al 2021) allow recording trajectories that are re-played in a loop. Visualisations of sound collections can be seen as a way of creating a terrain for live-coded sound-generating agents.…”

“…For large databases, the mapping of samples to the terrain should follow some logic. This can be based on audio descriptors, either directly (e.g., as in Schwarz et al 2006) or using dimensionality reduction (as in Roma et al 2021). Audio descriptors can also be used to visually represent each of the patches.…”