3D gaze cursor: Continuous calibration and end-point grasp control of robotic actuators

Tostado, Pablo M.; Abbott, Will; Faisal, A. Aldo

doi:10.1109/icra.2016.7487502

Cited by 21 publications

(22 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, instead we solve the Midas Touch Problem using a binocular eye-tracker to detect wink-based commands, thereby eliminating dwell times or long blinking. We previously demonstrated that binocular eye-tracking control enables real-time closed-loop control that outperforms invasive (and non-invasive) brainmachine interfaces in terms of cost and read-out data rates [12] for continuous 3D end-point control of robot actuators [25] and/or for free-gaze navigation of wheel-chairs [26].…”

Section: Discussionmentioning

confidence: 99%

“Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves

Noronha

Dziemian

Zito

et al. 2017

2017 International Conference on Rehabilitation Robotics (ICORR)

Self Cite

View full text Add to dashboard Cite

The ability of robotic rehabilitation devices to support paralysed end-users is ultimately limited by the degree to which human-machine-interaction is designed to be effective and efficient in translating user intention into robotic action. Specifically, we evaluate the novel possibility of binocular eye-tracking technology to detect voluntary winks from involuntary blink commands, to establish winks as a novel low-latency control signal to trigger robotic action. By wearing binocular eye-tracking glasses we enable users to directly observe their environment or the actuator and trigger movement actions, without having to interact with a visual display unit or user interface. We compare our novel approach to two conventional approaches for controlling robotic devices based on electromyo-graphy (EMG) and speech-based human-computer interaction technology. We present an integrated software framework based on ROS that allows transparent integration of these multiple modalities with a robotic system. We use a soft-robotic SEM glove (Bioservo Technologies AB, Sweden) to evaluate how the 3 modalities support the performance and subjective experience of the end-user when movement assisted. All 3 modalities are evaluated in streaming, closed-loop control operation for grasping physical objects. We find that wink control shows the lowest error rate mean with lowest standard deviation of (0.23 ± 0.07, mean ± SEM) followed by speech control (0.35 ± 0. 13) and EMG gesture control (using the Myo armband by Thalamic Labs), with the highest mean and standard deviation (0.46 ± 0.16). We conclude that with our novel own developed eye-tracking based approach to control assistive technologies is a well suited alternative to conventional approaches, especially when combined with 3D eye-tracking based robotic end-point control.

show abstract

Section: Discussionmentioning

confidence: 99%

“Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves

Noronha

Dziemian

Zito

et al. 2017

2017 International Conference on Rehabilitation Robotics (ICORR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…These results sit at one end of the spectrum of solutions for controlling an augmentative device, which goes from substitution all the way to direct augmentation via higher level control, either brain-machine-interfacing or cognitive interfaces such as eye gaze decoding. We previously showed that the end-point of visual attention (where one looks) can control the spatial end-point of a robotic actuator with centimetre-level precision (Tostado et al, 2016;Maimon-Mor et al, 2017;Shafti et al, 2019b). This direct control modality is more effective from a user perspective than voice or neuromuscular signals as a natural control interface (Noronha et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Playing the piano with a robotic third thumb: Assessing constraints of human augmentation

Shafti

Haar

Mio

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We wanted to study the ability of our brains and bodies to be augmented by supernumerary robot limbs, here extra fingers. We developed a mechanically highly functional supernumerary robotic 3 rd thumb actuator, the SR3T, and interfaced it with human users enabling them to play the piano with 11 fingers. We devised a set of measurement protocols and behavioural "biomarkers", the Human Augmentation Motor Coordination Assessment (HAMCA), which allowed us a priori to predict how well each individual human user could, after training, play the piano with a two-thumbs-hand. To evaluate augmented music playing ability we devised a simple musical score, as well as metrics for assessing the accuracy of playing the score. We evaluated the SR3T (supernumerary robotic 3 rd thumb) on 12 human subjects including 6 naïve and 6 experienced piano players. We demonstrated that humans can learn to play the piano with a 6-fingered hand within one hour of training. For each subject we could predict individually, based solely on their HAMCA performance before training, how well they were able to perform with the extra robotic thumb, after training (training end-point performance). Our work demonstrates the feasibility of robotic human augmentation with supernumerary robotic limbs within short time scales. We show how linking the neuroscience of motor learning with dexterous robotics and human-robot interfacing can be used to inform a priori how far individual motor impaired patients or healthy manual workers could benefit from robotic augmentation solutions.

show abstract

“…In our case, we demonstrated that our FastOrient algorithm works out-out-of-the-box which does not require a complex training learning pipeline and is sufficient to allow a natural wrist and grasp interaction. With the technology that we have demonstrated, we should be able to use gaze-based 3D positioning of the arm for grasping, as demonstrated in [4]- [6], wink-based detection of grasping intention [7], and now automatic processing of the grasp orientation based on the findings of this paper, without the requirement for the user to fine-tune the orientation of the hand, resulting in a full, intuitive assistive system. We have demonstrated FastOrient across 5 different realistic surfaces and for 26 different objects pertaining to standard activities of daily living.…”

Section: Discussionmentioning

confidence: 99%

“…Multi-modal systems relying on unaffected abilities, e.g. the use of gaze-based robotic end-point control for reaching assistance [4]- [6], or the use wearable robotics controlled through eye winks and voice [7]. In all these cases, a suitable eventual grasp is only possible with the correct orientation of the hand; be it the human hand using an orthotic ( Figure 1-left), or a tele-operated robotic hand, performing the grasp (Figure 1-right).…”

Section: Introductionmentioning

confidence: 99%

Fast orient: lightweight computer vision for wrist control in assistive robotic grasping

Maymo¹,

Shafti

Amir³

2018

2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob)

View full text Add to dashboard Cite

Wearable and Assistive robotics for human grasp support are broadly either tele-operated robotic arms or act through orthotic control of a paralyzed user's hand. Such devices require correct orientation for successful and efficient grasping. In many human-robot assistive settings, the end-user is required to explicitly control the many degrees of freedom making effective or efficient control problematic. Here we are demonstrating the off-loading of low-level control of assistive robotics and active orthotics, through automatic end-effector orientation control for grasping. This paper describes a compact algorithm implementing fast computer vision techniques to obtain the orientation of the target object to be grasped, by segmenting the images acquired with a camera positioned on top of the end-effector of the robotic device. The rotation needed that optimises grasping is directly computed from the object's orientation. The algorithm has been evaluated in 6 different scene backgrounds and end-effector approaches to 26 different objects. 94.8% of the objects were detected in all backgrounds. Grasping of the object was achieved in 91.1% of the cases and has been evaluated with a robot simulator confirming the performance of the algorithm.

show abstract

3D gaze cursor: Continuous calibration and end-point grasp control of robotic actuators

Cited by 21 publications

References 16 publications

“Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves

“Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves

Playing the piano with a robotic third thumb: Assessing constraints of human augmentation

Fast orient: lightweight computer vision for wrist control in assistive robotic grasping

Contact Info

Product

Resources

About