Wormhole Learning

Zanardi, Alessandro; Zilly, Julian; Aumiller, Andreas Jianhao; Censi, Andrea

doi:10.1109/icra.2019.8794336

Cited by 3 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To our knowledge, these extended intertwined notions of observability and data sufficiency for multiple sensors are not well explored, neither in robotics, nor in the broader learning/detection/filtering fields. While in [1], the transfer learning region 3 is taken for granted, in this work we look into how to enforce this region via cross-modal learning filters (XLFs). Symmetrically, during the first domain transfer we enforce the samples to be drawn from 3 rather than 2, during the backward transfer, we ensure samples are not taken from 5 but only from 4 or 3.…”

Section: Related Workmentioning

confidence: 99%

“…There are two sensors available. The work [1] considered the case of an RGB camera and an IR camera; the present work considers the pair of an RGB camera with an event-based neuromorphic sensor. The data from sensor a is a random variable Z a which takes values z a in set Z a .…”

Section: A Assumptionsmentioning

confidence: 99%

“…In an intuitive sense, the sensors must have operating domains that are distinct enough to not be redundant, but overlapping enough to allow transfer of some information. Refer to [1] for an information-theoretic formalization of this idea.…”

Section: A Assumptionsmentioning

confidence: 99%

“…For the particular case reported in [1], the two sensors were an RGB camera and an infrared (IR) camera, the learning methods were convolutional neural networks (CNNs) [15], and the initial approximation was a pre-trained neural network that was trained only on daytime data. The algorithm above was specified as follows: 1) Obtain a joint dataset for RGB and IR camera.…”

Section: Example Applied To Rgb and Ir Camerasmentioning

confidence: 99%

“…Wormhole learning [1] is a technique that can be used in an object detection scenario to enlarge the effective operating domain of a detector, if one has an auxiliary sensor available temporarily. One can let sensor 1 "teach" sensor 2; then, the student becomes the teacher, and sensor 2 "teaches" sensor 1 as illustrated in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Cross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning

Zanardi¹,

Aumiller²,

Zilly³

et al. 2019

Robotics: Science and Systems XV

Self Cite

View full text Add to dashboard Cite

Robots that need to act in an uncertain, populated, and varied world need heterogeneous sensors to be able to perceive and act robustly. For example, self-driving cars currently on the road are equipped with dozens of sensors of several types (lidar, radar, sonar, cameras, . . . ). All of this existing and emerging complexity opens up many interesting questions regarding how to deal with multi-modal perception and learning.The recently developed technique of "wormhole learning" shows that even temporary access to a different sensor with complementary invariance characteristics can be used to enlarge the operating domain of an existing object detector without the use of additional training data. For example, an RGB object detector trained with daytime data can be updated to function at night time by using a "wormhole" jump through a different modality that is more illumination invariant, such as an IR camera. It turns out that having an additional sensor improves performance, even if you subsequently lose it.In this work we extend wormhole learning to allow it to cope with sensors that are radically different, such as RGB cameras and event-based neuromorphic sensors. Their profound differences imply that we need a more careful selection of which samples to transfer, thus we design "cross-modal learning filters". We will walk in a relatively unexplored territory of multi-modal observability that is not usually considered in machine learning. We show that wormhole learning increases performance even though the intermediate neuromorphic modality is on average much worse at the task. These results suggest that multi-modal learning for perception is still an early field and there might be many opportunities to improve the perception performance by accessing a rich set of heterogeneous sensors (even if some are not actually deployed on the robot).

show abstract

Section: Related Workmentioning

confidence: 99%

Section: A Assumptionsmentioning

confidence: 99%

Section: A Assumptionsmentioning

confidence: 99%

Section: Example Applied To Rgb and Ir Camerasmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Cross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning

Zanardi¹,

Aumiller²,

Zilly³

et al. 2019

Robotics: Science and Systems XV

Self Cite

View full text Add to dashboard Cite

show abstract

Bridging the Gap Between Events and Frames Through Unsupervised Domain Adaptation

Nico

Gehrig

et al. 2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Reliable perception during fast motion maneuvers or in high dynamic range environments is crucial for robotic systems. Since event cameras are robust to these challenging conditions, they have great potential to increase the reliability of robot vision. However, event-based vision has been held back by the shortage of labeled datasets due to the novelty of event cameras. To overcome this drawback, we propose a task transfer method to train models directly with labeled images and unlabeled event data. Compared to previous approaches, (i) our method transfers from single images to events instead of high frame rate videos, and (ii) does not rely on paired sensor data. To achieve this, we leverage the generative event model to split event features into content and motion features. This split enables efficient matching between latent spaces for events and images, which is crucial for successful task transfer. Thus, our approach unlocks the vast amount of existing image datasets for the training of event-based neural networks. Our task transfer method consistently outperforms methods targeting Unsupervised Domain Adaptation for object detection by 0.26 mAP (increase by 93%) and classification by 2.7% accuracy.

show abstract

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception

Chen

Cao

Conradt

et al. 2020

IEEE Signal Process. Mag.

190

View full text Add to dashboard Cite

s a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.

show abstract

Wormhole Learning

Cited by 3 publications

References 35 publications

Cross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning

Cross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning

Bridging the Gap Between Events and Frames Through Unsupervised Domain Adaptation

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception

Contact Info

Product

Resources

About