MoMusic: A Motion-Driven Human-AI Collaborative Music Composition and Performing System

Abstract

The significant development of artificial neural network architectures has facilitated the increasing adoption of automated music composition models over the past few years. However, most existing systems feature algorithmic generative structures based on hard code and predefined rules, generally excluding interactive or improvised behaviors. We propose a motion based music system, MoMusic, as a AI real time music generation system. MoMusic features a partially randomized harmonic sequencing model based on a probabilistic analysis of tonal chord progressions, mathematically abstracted through musical set theory. This model is presented against a dual dimension grid that produces resulting sounds through a posture recognition mechanism. A camera captures the users' fingers' movement and trajectories, creating coherent, partially improvised harmonic progressions. MoMusic integrates several timbrical registers, from traditional classical instruments such as the piano to a new ''human voice instrument'' created using a voice conversion technique. Our research demonstrates MoMusic's interactiveness, ability to inspire musicians, and ability to generate coherent musical material with various timbrical registers. MoMusic's capabilities could be easily expanded to incorporate different forms of posture controlled timbrical transformation, rhythmic transformation, dynamic transformation, or even digital sound processing techniques.

Virtual Human: A Comprehensive Survey on Academic and Applications

“…Computer vision-based motion capture is currently less accurate, but has low environmental requirements, a large movable range, imaginative use of scenarios, and consumer-level applications. Bian et al introduce a motiondriven human-ai collaborative music composition and performance system, with a focus on improving the naturalness and creativity of music composition and performance [58]. Wang et al propose a method for motiondriven tracking via end-to-end coarse-to-fine verification, with a focus on improving the accuracy and stability of tracking [59].…”

Transparency in Music-Generative AI: A Systematic Literature Review

“…Both Louie et al (2020) and Arriagada (2020) state the power of AI as a co-creation tool, to help artists and enhance human creativity, rather than an threat to substitute artists. For instance, a couple of recently presented collaborative AI-human tool for music composition are MoMusic (Bian et al, 2023) and Humming2Music (Qiu et al, 2023). Furthermore, Hitsuwari et al ( 2023) presents a case study on hiku poetry were AI-human collaboration enhances beauty rating, compared with only human or only AI works.…”