Gaze, Visual, Myoelectric, and Inertial Data of Grasps for Intelligent Prosthetics

Cognolato, Matteo; Gijsberts, Arjan; Gregori, Valentina; Saetta, Gianluca; Giacomino, Katia; Hager, Anne-Gabrielle Mittaz; Gigli, Andrea; Faccio, Diego; Tiengo, Cesare; Bassetto, Franco; Caputo, Barbara; Brugger, Peter; Atzori, Manfredo; Müller, Henning

doi:10.1101/19010199

Cited by 5 publications

(7 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MeganePro dataset was acquired with the aim of investigating the use of gaze and visual information to improve prosthetic control (Cognolato et al, 2019). It contains data of 15 transradial amputees [13 M, 2 F; age: (47.13 ± 14.16) years] and a frequency matched control group of 30 able-bodied subjects [27 M, 3 F; age: (46.63 ± 15.11) years] who performed grasps and manipulation tasks with a variety of household items.…”

Section: Methodsmentioning

confidence: 99%

“…The first fixation is defined as the first of at least two successive samples where the gaze-target distance is <20 px. This threshold was chosen to accommodate for some systematic error in the gaze tracking and is roughly twice the average gaze tracking accuracy (Cognolato et al, 2019). The requirement for two successive samples that fall below the threshold is to ignore occasional outliers.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

Gregori¹,

Cognolato²,

Saetta³

et al. 2019

Front. Bioeng. Biotechnol.

Self Cite

View full text Add to dashboard Cite

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

Gregori¹,

Cognolato²,

Saetta³

et al. 2019

Front. Bioeng. Biotechnol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…While this study found that vision had an impact on predicting grasp poses, it is unknown whether gaze has the same impact on solely the wrist rotations. Cognolato et al [23] provides an approach towards grasp classification by using vision and gaze tracking on amputees that already use prosthetic hands in order to provide an improvement in creating pre-grasp hand poses. A large dataset was collected on both intact subjects and amputees, and provides gaze data, as well as myoelectric signals, and various learning algorithms were implemented for grasp classification.…”

Section: B Current Literaturementioning

confidence: 99%

Improving Automatic Control of Upper-Limb Prosthesis Wrists Using Gaze-Centered Eye Tracking and Deep Learning

Karrenbach

Boe

Sie

et al. 2022

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Many upper-limb prostheses lack proper wrist rotation functionality, leading to users performing poor compensatory strategies, leading to overuse or abandonment. In this study, we investigate the validity of creating and implementing a data-driven predictive control strategy in object grasping tasks performed in virtual reality. We propose the idea of using gazecentered vision to predict the wrist rotations of a user and implement a user study to investigate the impact of using this predictive control. We demonstrate that using this vision-based predictive system leads to a decrease in compensatory movement in the shoulder, as well as task completion time. We discuss the cases in which the virtual prosthesis with the predictive model implemented did and did not make a physical improvement in various arm movements. We also discuss the cognitive value in implementing such predictive control strategies into prosthetic controllers. We find that gaze-centered vision provides information about the intent of the user when performing object reaching and that the performance of prosthetic hands improves greatly when wrist prediction is implemented. Lastly, we address the limitations of this study in the context of both the study itself as well as any future physical implementations.

show abstract

“…Meanwhile, the skin wrapping around two bones realizes the tactile perception to the environment. Decades of efforts have brought about great progress on intelligent bionic prosthetic hand: in order to build the visual sensation substitution, the method of multimodal information fusion based on computer vision, inertial [3] and eyes tracking [4] was proposed to assist the manipulating of prosthetic hand. With the rich perception of RGBD camera and the excellent recognition of deep learning, the method of computer vision improved the grasping accuracy in the daily life scenarios [5,6].…”

Section: Introductionmentioning

confidence: 99%