Audio Visual Attention Models in the Mobile Robots Navigation

Pleshkova, Snejana; Bekiarski, Alexander

doi:10.1007/978-3-319-32192-9_8

Cited by 4 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Audio-visual correspondence also allows audio signals to serve as supervision for visual learning [24]. In the context of robotics and situated interactions, joint reasoning using audio-visual inputs supports target localization, tracking, and hence navigation (e.g., [25]- [28]). For example, a deep neural network that utilizes both vision and auditory inputs outperforms conventional methods that rely on single modalities in object tracking [25].…”

Section: Audio-visual Joint Reasoningmentioning

confidence: 99%

Object Permanence Through Audio-Visual Representations

Bu¹,

Huang

2021

IEEE Access

View full text Add to dashboard Cite

As robots perform manipulation tasks and interact with objects, it is probable that they accidentally drop objects that subsequently bounce out of their visual fields (e.g., due to an inadequate grasp of an unfamiliar object). To enable robots to recover from such errors, we draw upon the concept of object permanence-objects remain in existence even when they are not being sensed (e.g., seen) directly. In particular, we developed a multimodal neural network model-using a partial, observed bounce trajectory and the audio resulting from drop impact as its inputs-to predict the full bounce trajectory and the end location of a dropped object. We empirically show that: (1) our multimodal method predicted end locations close in proximity (i.e., within the visual field of the robot's wrist camera) to the actual locations and (2) the robot was able to retrieve dropped objects by applying minimal vision-based pick-up adjustments. Additionally, we show that our method outperformed five comparison baselines in retrieving dropped objects. Our results contribute to enabling object permanence for robots and error recovery from object drops.

show abstract

Section: Audio-visual Joint Reasoningmentioning

confidence: 99%

Object Permanence Through Audio-Visual Representations

Bu¹,

Huang

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Joint reasoning using audio-visual inputs, in general, supports target localization, tracking, and hence navigation (e.g., [18]- [21]). For example, a deep neural network that utilizes both vision and auditory inputs outperforms conventional methods that rely on single modalities in object tracking [18].…”

Section: Audio-visual Joint Reasoningmentioning

confidence: 99%

Object Permanence Through Audio-Visual Representations

Bu,

Huang

2020

Preprint

View full text Add to dashboard Cite

As robots perform manipulation tasks and interact with objects, it is probable that they accidentally drop objects that subsequently bounce out of their visual fields (e.g., due to an inadequate grasp of an unfamiliar object). To enable robots to recover from such errors, we draw upon the concept of object permanence-objects remain in existence even when they are not being sensed (e.g., seen) directly. In particular, we developed a multimodal neural network model-using a partial, observed bounce trajectory and the audio resulting from drop impact as its inputs-to predict the full bounce trajectory and the end location of a dropped object. We empirically show that: (1) our multimodal method predicted end locations close in proximity (i.e., within the visual field of the robot's wrist camera) to the actual locations and (2) the robot was able to retrieve dropped objects by applying minimal vision-based pick-up adjustments. Additionally, we show that our multimodal method outperformed the vision-only and audio-only baselines in retrieving dropped objects. Our results provide insights in enabling object permanence for robots and offer foundations for ensuring robust robot autonomy in task execution.

show abstract

Innovative Algorithms in Computer Vision

Jain

Favorskaya

2017

Intelligent Systems Reference Library

View full text Add to dashboard Cite

Audio Visual Attention Models in the Mobile Robots Navigation

Cited by 4 publications

References 32 publications

Object Permanence Through Audio-Visual Representations

Object Permanence Through Audio-Visual Representations

Object Permanence Through Audio-Visual Representations

Innovative Algorithms in Computer Vision

Contact Info

Product

Resources

About