Handmate: An Accessible Browser-based Controller for Web Audio and Midi using AI Hand-Tracking

Abstract

Handmate is a browser-based hand gestural controller for Web Audio and MIDI using open-source pose estimation technology from Google MediaPipe. The web browser was chosen as the platform to execute all computational processes so that users can access the latest computer vision technology for music-making without requiring any specialised hardware, software or technical expertise. Its innovation lies in combining pre-existing technologies together in a browser environment to enable accessible interactive sound-movement-making using hand movement. Two controllers are described. Handmate Effects, which receives microphone or user sound input and provides a choice of audio effects which can be controlled by hand movement, and Handmate MIDI, which outputs MIDI notes or control values.