3D interaction techniques for musical expression

Abstract

As Virtual Reality headsets become accessible, more and more artistic applications are developed, including immersive musical instruments. 3D interaction techniques designed in the 3D User Interfaces research community, such as navigation, selection and manipulation techniques, open numerous opportunities for musical control. For example, navigation techniques such as teleportation, free walking/flying and path-planning enable different ways of accessing musical scores, scenes of spatialized sounds sources or even parameter spaces. Manipulation techniques provide novel gestures and metaphors, e.g. for drawing or sculpting sound entities. Finally, 3D selection techniques facilitate the interaction with complex visual structures which can represent hierarchical temporal structures, audio graphs, scores or parameter spaces. However, existing devices and techniques were developed mainly with a focus on efficiency, i.e. minimising error rate and task completion times. They were therefore not designed with the specifics of musical interaction in mind. In this paper, we review existing 3D interaction techniques and examine how they can be used for musical control, including the possibilities they open for instrument designers. We then propose a number of research directions to adapt and extend 3DUIs for musical expression KEYWORDS 3D user interfaces, Immersive virtual musical instruments, Virtual Reality, New interface for musical expression

Music in Extended Realities

“…Loveridge offered a survey of the emerging field of networked music performances in VR through 2019 [89]. Berthaut reviewed existing 3D interaction techniques and examined how they can be used for musical control [14]. Atherton and Wang devised an overview of recent musical works that employ the VR medium [4], while Çamcı and Hamilton identified research trends in this field through a set of workshops that focus on Audio-first VR [34].…”

“…Berthaut raises an important point in [14], stating that existing 3D interaction techniques are not designed with musical interaction in mind, but are rather geared towards minimizing task completions times. Based on his review of existing techniques, the author proposes a number of directions to improve the usability of such techniques in musical applications.…”

“…Various hardware tools have also been conceived for Musical XR applications [14]. One such tool called Piivert, is an This work is licensed under a Creative Commons Attribution 4.0 License.…”

First Steps Towards Augmented Reality Interactive Electronic Music Production

“…For instance, many IMSIs have been developed, such as virtual xylophones or air guitar in room-scale environments [17], using three main interaction techniques, i.e. selection, manipulation and navigation [5]. About VR compositional frameworks, some of them allow to procedurally generate sounds with modular interfaces, enabling collaborative performances and improvisation [21,27].…”

Mixed Reality Interaction Techniques

“…While this chapter follows the above mentioned structure, further possibilities to structure interaction techniques include organizing according to interaction tasks [41] such as object selection [4,80,302] and object manipulation [219], navigation [163], symbolic input [89] or system control [79]. Further, interaction techniques for specific application domains have been discussed such as music [27] games [273] or immersive analytics [51,98]. Interested readers are also referred to further surveys on 3D interaction techniques [145] or interaction with smart glasses [196].…”