Playing the piano with a robotic third thumb: Assessing constraints of human augmentation

Abstract

We wanted to study the ability of our brains and bodies to be augmented by supernumerary robot limbs, here extra fingers. We developed a mechanically highly functional supernumerary robotic 3 rd thumb actuator, the SR3T, and interfaced it with human users enabling them to play the piano with 11 fingers. We devised a set of measurement protocols and behavioural "biomarkers", the Human Augmentation Motor Coordination Assessment (HAMCA), which allowed us a priori to predict how well each individual human user could, after training, play the piano with a two-thumbs-hand. To evaluate augmented music playing ability we devised a simple musical score, as well as metrics for assessing the accuracy of playing the score. We evaluated the SR3T (supernumerary robotic 3 rd thumb) on 12 human subjects including 6 naïve and 6 experienced piano players. We demonstrated that humans can learn to play the piano with a 6-fingered hand within one hour of training. For each subject we could predict individually, based solely on their HAMCA performance before training, how well they were able to perform with the extra robotic thumb, after training (training end-point performance). Our work demonstrates the feasibility of robotic human augmentation with supernumerary robotic limbs within short time scales. We show how linking the neuroscience of motor learning with dexterous robotics and human-robot interfacing can be used to inform a priori how far individual motor impaired patients or healthy manual workers could benefit from robotic augmentation solutions.

Human augmentation by wearable supernumerary robotic limbs: review and perspectives

“…Whether it is worn on the palm or on the wrist, the finger can increase the grasping capabilities of the human hand and its dexterity even in complex actions (e.g. playing piano [58]).…”

“…Horizontal movement controlled by the motion of the user's thumb of the augmented hand, vertical DoF controlled by the lifting of the left foot [57]. Both DoFs controlled by moving the foot [58]. Fin Ray sixth finger (2019), Hussain et al [59] Linear actuator for the bending motion.…”

“…The Third Thumb introduced by Cunningham et al [57], is controlled by two human motions, either coming from different body parts (thumb of the augmented hand and foot [57]), or coming both from the same limb (foot [58]). Pilot tests conducted with the third thumb showed that controlling the device demands high coordination and cognitive load at the beginning, but these aspects can be mitigated thanks to training [57].…”

“…The motion of the finger is proportional to the applied pressure. While the control interface presented in [58] is based on an IMU sensor worn on a single foot, the interfaces used in [3] employ pressure sensors worn underneath two different feet.…”

Highlights from the 30th Annual Meeting of the Society for the Neural Control of Movement

“…By comparing naïve participants and expert musicians using a foot-controlled supernumerary robotic thumb to play the piano, Aldo Faisal asked what determines our ability to learn and use augmentation in skilled tasks. They showed that foot dexterity (and not task-relevant piano expertise) is the best predictor of future performance (44). Additionally, the observation of highly idiosyncratic learning curves prompted new questions for future research: can everyone be augmented equally?…”