Hybrid Deblur Net: Deep Non-Uniform Deblurring With Event Camera

Abstract

Despite CNN-based deblur models have shown their superiority when solving motion blurs, restoring a photorealistic image from severe motion blur remains an ill-posed problem due to the loss of temporal information and textures. Event cameras such as Dynamic and Active-pixel Vision Sensor (DAVIS) [3] can simultaneously produce gray-scale Active Pixel Sensor (APS) frames and events, which can capture fast motions as events of very high temporal resolution, i. e., 1µs, can provide extra information for blurry APS frames. Due to the natural noise and sparsity of events, we employ a recurrent encoderdecoder architecture to generate dense recurrent event representations, which encode the overall historical information. We concatenate the original blurry image with the event representation as our hybrid input, from which the network learns to restore the sharp output. We conduct extensive experiments on GoPro dataset and a real event blurry dataset captured by DAVIS240C. Our experimental results on both synthetic and real images demonstrate state-of-the-art performance for 1280 × 720 images at 30 fps.INDEX TERMS Event-based Vision, High Speed, Image Deblurring, Real-Time.

Face Pose Alignment with Event Cameras

“…Hybrid pipelines: Hybrid of complementary event and image sensors can remove some of their shortcomings while enabling the benefits. There also exists compact hybrid sensor solutions [ 1 ] sharing the same lens, which can be employed, for instance, for motion de-blurring as shown recently [ 33 ]. Therefore, alternatively, for energy efficient higher precision and detailed alignment, the coarse face pose estimation can activate a more precise frame-based image acquisition and processing, after a first detection, implementing an approach similar to progressive initialization [ 43 ].…”

“…Tree ensembles have also been suggested in a very recent study [ 32 ] instead of DNNs for the similar efficiency reasons, though for a different vision problem. Although DNNs have also been applied with ECs for different vision problems [ 33 , 34 , 35 ], those studies aim at benefits of ECs other than the energy efficiency, thus are computationally very demanding. The comparison of our method to human performance in facial landmark placement on the same dataset shows that extremely randomized trees (ERT) cascade applied directly on pixel-events—i.e., without facial image reconstruction and training on image datasets—has good accuracy.…”

Fast Classification and Action Recognition With Event-Based Imaging

“…Several studies such as E2VID [20], EventSR [21] and HDN [22] reconstruct events back to normal images, and then the generated images can be processed similar to conventional vision tasks. The reconstruction methods usually utilize recursive modules such as ConvLSTM [23] with encoder-decoder structures such as UNet [24] to build images based on the events [20], [25].…”

Image Deblurring Aided by Low-Resolution Events

“…Pan et al [15] proposed an event-based double integral model for obtaining a high-framerate video from events and a blurry intensity image and Lin et al [14] implemented the physical model proposed by [15] as a neural network, which achieved a high performance in terms of video deblurring and interpolation. Moreover, Wang et al [28] unified denoising, deblurring, and super-resolution in one model by an event-enhanced degeneration model and Zhang et al [29] proposed a hybrid deblur net for image deblurring with learned event representation. These methods have shown their advantage of image enhancement.…”