The Cyborg Philharmonic: Synchronizing interactive musical performances between humans and machines

Abstract

Music offers a uniquely abstract way for the expression of human emotions and moods, wherein melodic harmony is achieved through a succinct blend of pitch, rhythm, tempo, texture, and other sonic qualities. The emerging field of “Robotic Musicianship” focuses on developing machine intelligence, in terms of algorithms and cognitive models, to capture the underlying principles of musical perception, composition, and performance. The capability of new-generation robots to manifest music in a human-like artistically expressive manner lies at the intersection of engineering, computers, music, and psychology; promising to offer new forms of creativity, sharing, and interpreting musical impulses. This manuscript explores how real-time collaborations between humans and machines might be achieved by the integration of technological and mathematical models from Synchronization and Learning, with precise configuration for the seamless generation of melody in tandem, towards the vision of human–robot symphonic orchestra. To explicitly capture the key ingredients of a good symphony—synchronization and anticipation—this work discusses a possible approach based on the joint strategy of: (i) Mapping— wherein mathematical models for oscillator coupling like Kuramoto could be used for establishing and maintaining synchronization, and (ii) Modelling—employing modern deep learning predictive models like Neural Network architectures to anticipate (or predict) future state changes in the sequence of music generation and pre-empt transitions in the coupled oscillator sequence. It is hoped that this discussion will foster new insights and research for better “real-time synchronized human-computer collaborative interfaces and interactions”.

Social robotics in music education: A systematic review

“…Finally, we would like to highlight that the studies under examination provided evidence of positive effects on students' learning in the educational context related to music. This is also noted in previous studies (Cádiz et al, 2021;Chakraborty et al, 2021;Savery et al, 2021). Nonetheless, these results must be interpreted very cautiously, due to the limitations to generalize them because the sample used is so small.…”

“…Research into robotics and music is enjoying a boom (Cádiz et al, 2021) in the broad field of music research (Ilari, 2020). "Robotic music" focuses on developing the intelligence of machines, in terms of algorithms and cognitive models, with the aim of capturing the underlying principles of music perception, composition, and playing (Chakraborty et al, 2021). The ability of robots to manifest music in an artistically expressive manner, like that of humans, lies in a multidisciplinary field between engineering, computing, music, and sociology.…”

“…Another case of an experience shared by robots and humans can be found in Vear (2021), whose investigation centered on technical solutions and the artistic potential of artificial-intelligencedriven robots that co-create with a human musician who improvises (the author) in real time. Similarly, Chakraborty et al (2021) delved deeper into one of the characteristics of musical performances in which musicians do not play rigidly, but play, move, and behave depending on the "feeling" of the music, and in tandem or according to non-verbal mutual gestures between them or from the main conductor. All of this requires an advanced level of cognitive operation for musical robots within such interactive, synchronized, and collaborative environments.…”

“…We must not forget that music is not only regarded as an artistic teaching which beauty lies in its actual manifestation but also has proved beneficial in multiple aspects. By a way of example, concerning health, it enhances wellbeing and reduces stress; as for learning, music motivates students and predisposes them to learn other subjects (Chakraborty et al, 2021). Be that as it may, music teaching is essential for the comprehensive training of students.…”

Recognition of musical beat and style and applications in interactive humanoid robot

“…The starting point of notes can be determined by looking for the peak of short-term energy difference. Then, two layers of judgment are designed to determine the endpoint and reduce the dependence on the threshold (Chakraborty et al, 2021 ; Feng et al, 2021 ).…”