Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People

Abstract

We have applied interactive machine learning (IML) to the creation and customisation of gesturally controlled musical interfaces in six workshops with people with learning and physical disabilities. Our observations and discussions with participants demonstrate the utility of IML as a tool for participatory design of accessible interfaces. This work has also led to a better understanding of challenges in end-user training of learning models, of how people develop personalised interaction strategies with different types of pre-trained interfaces, and of how properties of control spaces and input devices influence people's customisation strategies and engagement with instruments. This work has also uncovered similarities between the musical goals and practices of disabled people and those of expert musicians.

Mixed-Initiative Creative Interfaces

“…practice accessible and enjoyable to non-professional and even disabled user groups, with rich benefits to personal wellbeing and societal inclusion [5,6,[12][13][14].…”

“…Examples of MICIs outside games demonstrate the relevance of the approach to a broad range of creative contexts under active consideration by HCI researchers, including: urban design [16], sketching [15], interface design [13], prototyping musical instruments [14], or data visualization [18]. Indeed, with the current rise of AI, (semi-)autonomous systems, and conversational agents [19,20], human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.…”

Machine Learning as Meta-Instrument: Human-Machine Partnerships Shaping Expressive Instrumental Creation

“…Previous publications describe how I have used participatory design processes and surveys (Fiebrink et al 2010), workshops (Katan et al 2015), interviews , analysis of software logs , and reflection on my own work (Fiebrink et al 2009) to understand how people use Wekinator and why. This work all suggests that the most important benefits of Wekinator pertain to the way that it changes the design process, facilitating the creation of new kinds of instruments while also making design accessible to new people.…”

User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit

“…For instance, ML can facilitate the analysis of audio, visual, and sensor data. ML can support the creation of new systems for embodied interaction and expression, as well as provide new creative workflows for rapid prototyping and product customisation (Hartmann et al, 2007;Fiebrink, Cook, & Trueman, 2011;Katan, Grierson, & Fiebrink, 2015). This paper describes the development of new ML tools within the context of a European Commissionfunded "Innovation Action"-a joint effort between academic institutions and companies aimed at technology transfer and the production of new or improved products or services.…”