Decoding of human hand actions to handle missing limbs in Neuroprosthetics

Abstract

The only way we can interact with the world is through movements, and our primary interactions are via the hands, thus any loss of hand function has immediate impact on our quality of life. However, to date it has not been systematically assessed how coordination in the hand's joints affects every day actions. This is important for two fundamental reasons. Firstly, to understand the representations and computations underlying motor control "in-the-wild" situations, and secondly to develop smarter controllers for prosthetic hands that have the same functionality as natural limbs. In this work we exploit the correlation structure of our hand and finger movements in daily-life. The novelty of our idea is that instead of averaging variability out, we take the view that the structure of variability may contain valuable information about the task being performed. We asked seven subjects to interact in 17 daily-life situations, and quantified behavior in a principled manner using CyberGlove body sensor networks that, after accurate calibration, track all major joints of the hand. Our key findings are: (1) We confirmed that hand control in daily-life tasks is very low-dimensional, with four to five dimensions being sufficient to explain 80-90% of the variability in the natural movement data. (2) We established a universally applicable measure of manipulative complexity that allowed us to measure and compare limb movements across tasks. We used Bayesian latent variable models to model the low-dimensional structure of finger joint angles in natural actions. (3) This allowed us to build a naïve classifier that within the first 1000 ms of action initiation (from a flat hand start configuration) predicted which of the 17 actions was going to be executed-enabling us to reliably predict the action intention from very short-time-scale initial data, further revealing the foreseeable nature of hand movements for control of neuroprosthetics and tele operation purposes. (4) Using the Expectation-Maximization algorithm on our latent variable model permitted us to reconstruct with high accuracy (<5-6 • MAE) the movement trajectory of missing fingers by simply tracking the remaining fingers. Overall, our results suggest the hypothesis that specific hand actions are orchestrated by the brain in such a way that in the natural tasks of daily-life there is sufficient redundancy and predictability to be directly exploitable for neuroprosthetics.

Grammars of action in human behavior and evolution

“…This sample includes 5 sequences for which upper limb movements and manual joint angles were recorded as part of a previous study 14 , and 6 for which the tools and waste produced were analyzed and compared with actual Paleolithic artifacts from the Middle Pleistocene site of Boxgrove, UK 15 . Building on this and other previous research [14][15][16][17][18][19] , we focused our current study on archaeologically documented tool-making methods from the early and late Lower Paleolithic, a period that witnessed a nearly 3-fold increase in hominin brain size. This allows us to empirically address the over 100 years of theorizing linking increasingly complex tool-making with brain evolution and language origins 5,[20][21][22] .…”

Sparse Eigenmotions derived from daily life kinematics implemented on a dextrous robotic hand

“…Our findings suggest that the SEMs can be used to simplify the interaction of high-dimensional dextrous end-effectors utilising a lower dimensional control manifold defined by the SEM. We previously showed in neuroprosthetic hands that we can boost decoding accuracy of movement intention [11] if we apply the known spatio-temporal structure of kinematics [32][33][34][35].…”

“…Instead of using highly controlled stereotyped examples of human hand movements (e.g. repeated grasp of specific objects) we studied hand behaviour in a natural setting [1], [11], a studio flat environment in which subjects lived and worked day long and whose hand kinematics were collected using wearable sensors.…”

EthoHand: A dexterous robotic hand with ball-joint thumb enables complex in-hand object manipulation

“…Our hands are considered to be one of the most complex end-effectors comprised by various types of joints, that require the synergistic movement of 27 bones and co-activation of 37 [1] muscles in order to function and explain observed hand kinematics [2]. The human hand's 24+ degrees of freedom (DoFs) allow for dexterous and precise tasks to be performed, with the least amount of effort.…”

“…flint stone tool making [3]) or modern day activities (e.g. [2]). Emulating our hands' level of functionality in an anthropomorphic device has proven to be extremely difficult [5].…”