DA4Event: towards bridging the Sim-to-Real Gap for Event Cameras using Domain Adaptation

Abstract

Event cameras are novel bio-inspired sensors, which asynchronously capture pixel-level intensity changes in the form of "events". The innovative way they acquire data presents several advantages over standard devices, especially in poor lighting and high-speed motion conditions. However, the novelty of these sensors results in the lack of a large amount of training data capable of fully unlocking their potential. The most common approach implemented by researchers to address this issue is to leverage simulated event data. Yet, this approach comes with an open research question: how well simulated data generalize to real data? To answer this, we propose to exploit, in the event-based context, recent Domain Adaptation (DA) advances in traditional computer vision, showing that DA techniques applied to event data help reduce the sim-to-real gap. To this purpose, we propose a novel architecture, which we call Multi-View DA4E (MV-DA4E), that better exploits the peculiarities of frame-based event representations while also promoting domain invariant characteristics in features.Through extensive experiments, we prove the effectiveness of DA methods and MV-DA4E on N-Caltech101. Moreover, we validate their soundness in a real-world scenario through a cross-domain analysis on the popular RGB-D Object Dataset (ROD), which we extended to the event modality (RGB-E).*This paper is currently under review. It was partially supported by the ERC project RoboExNovo. Computational resources were partially provided by IIT.

E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

“…Discussion and Limitations. As it is currently not possible to fully replicate event camera behaviors, event simulation may create undesirable sim-to-real domain shift [78,90]. Nevertheless, several works showed that simulated events are robust enough to generalize well to real ones [33,78,90].…”

“…As it is currently not possible to fully replicate event camera behaviors, event simulation may create undesirable sim-to-real domain shift [78,90]. Nevertheless, several works showed that simulated events are robust enough to generalize well to real ones [33,78,90]. As we introduce event data in egocentric action recognition for the first time, we aim at providing a direct comparison with common benchmarks in the literature [14,15,27] and place the event modality in a competitive setting against well-established modalities.…”

“…While traditional cameras are capable of providing very rich visual information at the tradeoff of slow and often redundant updates, event-based cameras are asynchronous and spatially sparse, and capable of microseconds temporal resolution. Event-based systems range from designs that focus on exploiting and maintaining event-camera sparsity during computation [4,85,107], to algorithms that combine events with standard cameras [7,35,46,78,99], exploiting the complementarity of the two. With the goal of achieving minimum-delay computing, research has also focused on asynchronous designs, either by modifying regular CNNs [5,69] or by utilizing specific hardware solutions [2,21,29], often leveraging on bio-inspired computing frameworks [68].…”

“…Then, as the performance of different modalities may greatly vary depending on the architecture used for processing [79], we benchmark events using three of the most accredited architectures in FPAR, namely TSM [62], TSN [103] and I3D [8]. We leverage a well-established procedure for converting event streams into a frame-like representation that has been shown to efficiently integrate with off-the-shelf CNNs [78,90], and finally propose to encourage modeling of motion features by employing attention at channel level.…”

“…Frame-like representations are by far the most widespread methods as they can be directly used together with off-the-shelf networks. Among available ones [5,6,25,34,48,55,86,113] we chose Voxel Grid [113] as it proved to be superior in cross-domain settings [78,90]. This representation computes a B-channel image by discretizing time in B separate intervals: Backbone Architectures.…”