Estimating finger grip force from an image of the hand using Convolutional Neural Networks and Gaussian processes

Abstract

Abstract-Estimating human fingertip forces is required to understand force distribution in grasping and manipulation. Human grasping behavior can then be used to develop forceand impedance-based grasping and manipulation strategies for robotic hands. However, estimating human grip force naturally is only possible with instrumented objects or unnatural gloves, thus greatly limiting the type of objects used.In this paper we describe an approach which uses images of the human fingertip to reconstruct grip force and torque at the finger. Our approach does not use finger-mounted equipment, but instead a steady camera observing the fingers of the hand from a distance. This allows for finger force estimation without any physical interference with the hand or object itself, and is therefore universally applicable.We construct a 3-dimensional finger model from 2D images. Convolutional Neural Networks (CNN) are used to predict the 2D image to a 3D model transformation matrix. Two methods of CNN are designed for separate and combined outputs of orientation and position. After learning, our system shows an alignment accuracy over 98% on unknown data.In the final step, a Gaussian process estimates finger force and torque from the aligned images based on color changes and deformations of the nail and its surrounding skin. Experimental results shows that the accuracy achieves about 95% in the force estimation and 90% in the torque.

On-field player workload exposure and knee injury risk monitoring via deep learning

“…velocity) based on gross assumptions of linear regression, which overfit to a simple movement pattern or participant cohort [7], however, researchers have reported success deriving kinematics from these devices for movement classification [40,52]. To improve on these methods, a number of research teams have sought to leverage computer vision and data science techniques, and while initial results appear promising, to date they lack validation to ground truth data, or relevance to specific sporting related tasks [8,49,53]. For example, Fluit et al [22] and Yang et al [57] derive GRF/M from motion capture.…”

Predicting Athlete Ground Reaction Forces and Moments From Spatio-Temporal Driven CNN Models

“…Previous computer vision and data science researchers have attempted to estimate GRF/Ms, however studies suffer from poor validation to ground truth data, are not sports related [17], [18], [19], require a full body modeling protocol with multiple inputs [20], or as before, predict only unidirectional GRF components (e.g. vertical F z ) [21].…”

Estimating Fingertip Forces, Torques, and Local Curvatures from Fingernail Images

“…13 [4] estimates force and torque using Gaussian processes (GP) and neural networks. Given the high accuracy obtained in [4][5][6] we use Gaussian processes to estimate force from the aligned images in this paper. We also explore other methods for the estimation such as Neural Networks, Convolutional Neural Networks and Recurrent Neural Networks.…”