Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis

Marković, Marko; Došen, Strahinja; Popović, Dejan B.; Gramlich, Bernhard; Farina, Dario

doi:10.1088/1741-2560/12/6/066022

Cited by 106 publications

(89 citation statements)

References 55 publications

(60 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These workaround techniques have emerged mainly because the promised EMG pattern recognition-based methods have not proved robust, or even feasible, for grasp classification clinically. The non-intuitiveness and shortcomings of the aforementioned approaches have encouraged the emergence of techniques that advocate utilisation of sensing modalities other than the conventional EMG signals, such as accelerometry or in general inertial measurements [14,25,26,57], RFID tags [28], artificial vision including standard cameras as well as Kinect [29][30][31][32][33][34]. In almost all multi-modal approaches to control limb prosthesis, it is argued that the incorporation of two or more sources of information can reduce the users' cognitive burden and enhance functionality in terms of accuracy.…”

Section: Discussionmentioning

confidence: 99%

“…Skin movement analysis via accelerometry signals [25,26], force myography [27], use of radio-frequency identification (RFID) tags [28], arm movement trajectory and inertial measurement (e.g. i-mo TM ) and computer vision [29][30][31][32][33][34] are some examples. Specifically, in the case of using computer vision, it was shown that object shapes can be quantised such that appropriate grasp types and sizes can be determined.…”

Section: Introductionmentioning

confidence: 99%

“…Marković et al [33] further exploited a data fusion technique to control a prosthetic hand. A plethora of modalities, namely, myoelectric recording, computer vision, inertial measurements and embedded prosthesis sensors (position and force) were utilised to provide realtime simultaneous, proportional and semi-autonomous control.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep learning-based artificial vision for grasp classification in myoelectric hands

et al. 2017

View full text Add to dashboard Cite

Objective. Computer vision-based assistive technology solutions can revolutionise the quality of care for people with sensorimotor disorders. The goal of this work was to enable trans-radial amputees to use a simple, yet efficient, computer vision system to grasp and move common household objects with a two-channel myoelectric prosthetic hand. Approach. We developed a deep learning-based artificial vision system to augment the grasp functionality of a commercial prosthesis. Our main conceptual novelty is that we classify objects with regards to the grasp pattern without explicitly identifying them or measuring their dimensions. A convolutional neural network (CNN) structure was trained with images of over 500 graspable objects. For each object, 72 images, at intervals, were available. Objects were categorised into four grasp classes, namely: pinch, tripod, palmar wrist neutral and palmar wrist pronated. The CNN setting was first tuned and tested offline and then in realtime with objects or object views that were not included in the training set. Main results. The classification accuracy in the offline tests reached for the seen and for the novel objects; reflecting the generalisability of grasp classification. We then implemented the proposed framework in realtime on a standard laptop computer and achieved an overall score of in classifying a set of novel as well as seen but randomly-rotated objects. Finally, the system was tested with two trans-radial amputee volunteers controlling an i-limb UltraTM prosthetic hand and a motion controlTM prosthetic wrist; augmented with a webcam. After training, subjects successfully picked up and moved the target objects with an overall success of up to . In addition, we show that with training, subjects’ performance improved in terms of time required to accomplish a block of 24 trials despite a decreasing level of visual feedback. Significance. The proposed design constitutes a substantial conceptual improvement for the control of multi-functional prosthetic hands. We show for the first time that deep-learning based computer vision systems can enhance the grip functionality of myoelectric hands considerably.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning-based artificial vision for grasp classification in myoelectric hands

et al. 2017

View full text Add to dashboard Cite

show abstract

“…These methods are useful in simple and short tasks. However, as the number of functions increases, more time is needed to select the intended function [4,5]. Cocontraction switching is commonly used for switching between functions or prehensile types in multiarticulated hand systems or powered transhumeral prostheses.…”

Section: Introductionmentioning

confidence: 99%

Networked Multimodal Sensor Control of Powered 2-DOF Wrist and Hand

Shibuya

Ohnishi

Kajitani

2017

Journal of Robotics

View full text Add to dashboard Cite

A prosthetic limb control system to operate powered 2-DOF wrist and 1-DOF hand with environmental information, myoelectric signal, and forearm posture signal is composed and evaluated. Our concept model on fusing biosignal and environmental information for easier manipulation with upper limb prosthesis is assembled utilizing networking software and prosthetic component interlink platform. The target is to enhance the controllability of the powered wrist's orientation by processing the information to derive the joint movement in a physiologically appropriate manner. We applied a manipulative skill model of prehension which is constrained by forearm properties, grasping object properties, and task. The myoelectric and forearm posture sensor signals were combined with the work plane posture and the operation mode for grasping object properties. To verify the reduction of the operational load with the proposed method, we conducted 2 performance tests: system performance test to identify the powered 2-DOF wrist's tracking performance and user operation tests. From the system performance experiment, the fusion control was confirmed to be sufficient to control the wrist joint with respect to the work plane posture. Forearm posture angle ranges were reduced when the prosthesis was operated companying environmental information in the user operation tests.

show abstract

“…Recently, computer vision methods have been introduced to achieve better accuracy when decoding the users intended action. For instance, researchers have shown that appropriate grasp types can be determined using shape features [33]. Ghazal et al [34] used a deep learning-based artificial vision system which classifies objects with regards to the grasp pattern without explicitly identifying them or measuring their dimensions.…”

Section: Previous Workmentioning

confidence: 99%

Human-in-the-loop assistive cyber physical system control using physiological signals

Quivira¹

View full text Add to dashboard Cite

Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis

Cited by 106 publications

References 55 publications

Deep learning-based artificial vision for grasp classification in myoelectric hands

Deep learning-based artificial vision for grasp classification in myoelectric hands

Networked Multimodal Sensor Control of Powered 2-DOF Wrist and Hand

Human-in-the-loop assistive cyber physical system control using physiological signals

Contact Info

Product

Resources

About