GPA-Teleoperation: Gaze Enhanced Perception-Aware Safe Assistive Aerial Teleoperation

Wang, Qianhao; He, Botao; Zhiren, Xun,; Xu, Chao; Gao, Fei

doi:10.1109/lra.2022.3153898

Cited by 10 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…What would be possible applications of attention-prediction based visual navigation outside of drone racing? One application may be assistive technology supporting human drone operators in hands-free drone operation [44,45], vision-based inspection [46,47], and search-and-rescue missions [48,49].…”

Section: Discussionmentioning

confidence: 99%

Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

Pfeiffer,

Wengeler,

Loquercio

et al. 2022

Preprint

View full text Add to dashboard Cite

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural networks' performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Our results show that attention-prediction based controllers outperform the baselines and are able to complete a challenging race track consistently with up to 88% success rate. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

show abstract

Section: Discussionmentioning

confidence: 99%

Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

Pfeiffer,

Wengeler,

Loquercio

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Gaze-based HCI in computer vision. With the development of gaze estimation and tracking [1,23,24], gaze has become one of the main interaction modes in HCI systems [3,25,26]. At the same time, gaze is more widely applied to cross-modal computer vision tasks.…”

Section: Related Workmentioning

confidence: 99%

An Accuracy Enhanced Vision Language Grounding Method Fused with Gaze Intention

Zhang,

Tu,

Zhang

et al. 2023

Electronics

View full text Add to dashboard Cite

Visual grounding aims to recognize and locate the target in the image according to human intention, which provides a new intelligent interaction idea and method for augmented reality (AR) and virtual reality (VR) devices. However, existing vision language grounding adopts language modals for visual grounding, but it performs ineffectively for images containing multiple similar objects. Gaze interaction is an important interaction mode in AR/VR devices, and it provides an advanced solution to the inaccurate vision language grounding cases. Based on the above questions and analysis, a vision language grounding framework fused with gaze intention is proposed. Firstly, we collect the manual gaze annotations using the AR device and construct a novel multi-modal dataset, RefCOCOg-Gaze, combining it with the proposed data augmentation methods. Secondly, an attention-based multi-modal feature fusion model is designed, providing a baseline framework for vision language grounding with gaze intention (VLG-Gaze). Through a series of precisely designed experiments, we analyze the proposed dataset and framework qualitatively and quantitatively. Comparing with the state-of-the-art vision language grounding model, our proposed scheme improves the accuracy by 5.3%, which indicates the significance of gaze fusion in multi-modal grounding tasks.

show abstract

Human-guided motion planner with perception awareness for assistive aerial teleoperation

Shi,

Luo,

Gao

et al. 2024

Advanced Robotics

View full text Add to dashboard Cite

GPA-Teleoperation: Gaze Enhanced Perception-Aware Safe Assistive Aerial Teleoperation

Cited by 10 publications

References 30 publications

Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

An Accuracy Enhanced Vision Language Grounding Method Fused with Gaze Intention

Human-guided motion planner with perception awareness for assistive aerial teleoperation

Contact Info

Product

Resources

About