Machine Learning as Meta-Instrument: Human-Machine Partnerships Shaping Expressive Instrumental Creation

Fiebrink, Rebecca

doi:10.1007/978-981-10-2951-6_10

Cited by 12 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such efforts towards "reframing machine learning workflows based on situated human working practices, and exploring the co-adaptation of humans and intelligent systems" [19] p.7.1, show that systems can operate in response to limited training samples presented in real-world contexts. Fiebrink, for example, has shown considerable success in building digital music instruments that can be trained and respond to real-time embodied interaction by users [18]. Indeed, we find possible starting points in object recognizers which allow vision impaired users to build their own training sets [41], [49].…”

Section: Salience In the Momentmentioning

confidence: 96%

The Care Work of Access

Bennett

Rosner

Taylor

2020

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Current approaches to AI and Assistive Technology (AT) often foreground task completion over other encounters such as expressions of care. Our paper challenges and complements such task-completion approaches by attending to the care work of access-the continual affective and emotional adjustments that people make by noticing and attending to one another. We explore how this work impacts encounters among people with and without vision impairments who complete tasks together. We find that bound up in attempts to get things done are concerns for one another and how well people are doing together. Reading this work through emerging disability studies and feminist STS scholarship, we account for two important forms of work that give rise to access: (1) mundane attunements and (2) noninnocent authorizations. Together these processes work as sensitizing concepts to help HCI scholars account for the ways that intelligent ATs both produce access while sometimes subverting people with disabilities.

show abstract

Section: Salience In the Momentmentioning

confidence: 96%

The Care Work of Access

Bennett

Rosner

Taylor

2020

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Interactive music systems are often divided into three stages: sensing, processing, and response, as shown in Figure 3 [67]. While this framework is simple, it provides a helpful division of concerns and has previously been used to frame DMI designs [22] including those using ML [27]. This framework highlights that electronic music systems, unlike most acoustic instruments, are modular.…”

Section: Prediction In Musical Interactionmentioning

confidence: 99%

Deep Predictive Models in Interactive Music

Martín¹,

Ellefsen²,

Tørresen³

2018

Preprint

View full text Add to dashboard Cite

Musical performance requires prediction to operate instruments, to perform in groups and to improvise. In this paper, we investigate how a number of digital musical instruments (DMIs), including two of our own, have applied predictive machine learning models that assist users by predicting unknown states of musical processes. We characterise these predictions as focussed within a musical instrument, at the level of individual performers, and between members of an ensemble. These models can connect to existing frameworks for DMI design and have parallels in the cognitive predictions of human musicians.We discuss how recent advances in deep learning highlight the role of prediction in DMIs, by allowing data-driven predictive models with a long memory of past states. The systems we review are used to motivate musical use-cases where prediction is a necessary component, and to highlight a number of challenges for DMI designers seeking to apply deep predictive models in interactive music systems of the future.

show abstract

“…The advent of electronic musical instruments including powerful computers has allowed experiments with instruments that are able to make intelligent use of the musical context in which they are used. This has been discussed since at least the early 1990s (Pressing, 1990 ), but has been extended in recent years with the development and popularity of accessible machine learning frameworks for understanding physical gestures in performance (Fiebrink, 2017 ). Artificial intelligence techniques can imbue a musical interface with a kind of self-awareness (Lewis et al, 2016 ; Nymoen et al, 2016 ), allowing them to act predictively, rather than in reaction to a performer.…”

Section: Introductionmentioning

confidence: 99%

Understanding Musical Predictions With an Embodied Interface for Musical Machine Learning

Martín

Glette

Nygaard

et al. 2020

Front. Artif. Intell.

View full text Add to dashboard Cite

Machine-learning models of music often exist outside the worlds of musical performance practice and abstracted from the physical gestures of musicians. In this work, we consider how a recurrent neural network (RNN) model of simple music gestures may be integrated into a physical instrument so that predictions are sonically and physically entwined with the performer's actions. We introduce EMPI, an embodied musical prediction interface that simplifies musical interaction and prediction to just one dimension of continuous input and output. The predictive model is a mixture density RNN trained to estimate the performer's next physical input action and the time at which this will occur. Predictions are represented sonically through synthesized audio, and physically with a motorized output indicator. We use EMPI to investigate how performers understand and exploit different predictive models to make music through a controlled study of performances with different models and levels of physical feedback. We show that while performers often favor a model trained on human-sourced data, they find different musical affordances in models trained on synthetic, and even random, data. Physical representation of predictions seemed to affect the length of performances. This work contributes new understandings of how musicians use generative ML models in real-time performance backed up by experimental evidence. We argue that a constrained musical interface can expose the affordances of embodied predictive interactions.

show abstract

Machine Learning as Meta-Instrument: Human-Machine Partnerships Shaping Expressive Instrumental Creation

Cited by 12 publications

References 10 publications

The Care Work of Access

The Care Work of Access

Deep Predictive Models in Interactive Music

Understanding Musical Predictions With an Embodied Interface for Musical Machine Learning

Contact Info

Product

Resources

About