Improving networked music performance systems using application‐network collaboration

Abstract

Summary Networked Music Performance (NMP) systems involve musicians located in different places who perform music while staying synchronized via the Internet. The maximum end‐to‐end delay in NMP applications is called Ensemble Performance Threshold (EPT) and should be less than 25 milliseconds. Due to this constraint, NMPs require ultra–low‐delay solutions for audio coding, network transmission, relaying, and decoding, each one a challenging task on its own. There are two directions for study in the related work referring to the NMP systems. From the audio perspective, researchers experiment on low‐delay encoders and transmission patterns, aiming to reduce the processing delay of the audio transmission, but they ignore the network performance. On the other hand, network‐oriented researchers try to reduce the network delay, which contributes to reduced end‐to‐end delay. In our proposed approach, we introduce an integration of dynamic audio and network configuration to satisfy the EPT constraint. The basic idea is that, the major components participating in an NMP system, the application and the network interact during the live music performance. As the network delay increases, the network tries to equalize it by modifying the routing behavior using Software‐Defined Networking principles. If the network delay exceeds a maximum affordable threshold, the network reacts by informing the application to change the audio processing pattern to overcome the delay increase, resulting in below EPT end‐to‐end delay. A full prototype of the proposed system was implemented and extensively evaluated in an emulated environment.

A Presence- and Performance-Driven Framework to Investigate Interactive Networked Music Learning Scenarios

“…While in music tuition, conflicts and breakdowns are managed with the time flexibility of turn-taking in conversation [33], remote playing demands more extreme responsivity. e immediacy of responses between remote performers is essentially altered by the presence of end-to-end delays, dependent on the signal processing, throughout the whole chain, and network delay [4,5]. e resulting latency level represents a factor that dramatically affects the NMP experience.…”

“…ese kinds of performance represent a proper task, which is essentially addressed to the external world and experienced from a second-person perspective, that is, a listener/observer. e concert is the con guration commonly investigated in NMP research and technological development [4,5,59]. Multimodal signal acquisition includes the audio recordings from both the remote environments, to compute several measures on the quality of the performance [19], gaze tracing and eye tracking to annotate the visual search of the musicians [60], and the biometric response of the performers for distress estimation [61].…”

“…Computer systems for networked musical interaction have been categorized according to their temporal (synchronous vs. asynchronous) and spatial (co-located vs. remote) dimensions of the performance [2], whereas a wide part of the scienti c literature has focused on the technological and perceptual issues in real-time performance between musicians located in remote rooms, and requiring the highest degree of synchronicity, typically over teleconference-based communication media [3][4][5]. From a di erent angle, for years, research in telematic arts and music has been questioning the NMP communication model, stressing instead the role of the body in the network, and proposing an interpretive model of the emerging space, wherein questions of presence, plausible representations, sense of agency, and ow become crucial [6][7][8].…”

Foreword to the Special Issue on the 2017 Edition of the Workshop on Performance Evaluation of communications in DIstributed Systems and WEb‐based Service Architectures (PEDISWESA 2017)

“…Finally, Lakiotakis et al propose an innovative solution for Networked Music Performance (NMP) systems that involve musicians located in different places who perform music while staying synchronized via the Internet by considering their challenging latency requirements, typically defined Ensemble Performance Threshold (EPT) that should be less than 25 ms. The authors propose an application‐aware approach and employ Software Defined Networking (SDN) as well as dynamic audio (re‐)configuration techniques to satisfy the EPT constraint.…”

“…The list of the papers of the special issue is provided . At this point, we want to thank all the people involved in the effort of producing this issue, starting from the submitting authors, to the team of enthusiastic reviewers who have provided useful suggestions to authors.…”