Learning Super-Resolution Reconstruction for High Temporal Resolution Spike Stream

Xiang, Xijie; Zhu, Lin; Li, Jianing; Wang, Yixuan; Huang, Tiejun

doi:10.1109/tcsvt.2021.3130147

Cited by 17 publications

(10 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As illustrated in Fig. 2(a), the input light is equally divided into Vidar and DAVIS346 via a beam splitter (i.e., Thorlabs CCM1-BS013) (Wang et al 2020c;Zhu et al 2021;Xiang et al 2021). On this basis, we design the spatio-temporal calibration procedures to synchronize two cameras within the shared view at the same time in Fig.…”

Section: Multimodal Neuromorphic Datasetmentioning

confidence: 99%

Retinomorphic Object Detection in Asynchronous Visual Streams

Wang

Zhu

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Due to high-speed motion blur and challenging illumination, conventional frame-based cameras have encountered an important challenge in object detection tasks. Neuromorphic cameras that output asynchronous visual streams instead of intensity frames, by taking the advantage of high temporal resolution and high dynamic range, have brought a new perspective to address the challenge. In this paper, we propose a novel problem setting, retinomorphic object detection, which is the first trial that integrates foveal-like and peripheral-like visual streams. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-Vidar-DVS) over 215.5k spatio-temporal synchronized labels. Then, we design temporal aggregation representations to preserve the spatio-temporal information from asynchronous visual streams. Finally, we present a novel bio-inspired unifying framework to fuse two sensing modalities via a dynamic interaction mechanism. Our experimental evaluation shows that our approach has significant improvements over the state-of-the-art methods with the single-modality, especially in high-speed motion and low-light scenarios. We hope that our work will attract further research into this newly identified, yet crucial research direction. Our dataset can be available at https://www.pkuml.org/resources/pku-vidar-dvs.html.

show abstract

Section: Multimodal Neuromorphic Datasetmentioning

confidence: 99%

Retinomorphic Object Detection in Asynchronous Visual Streams

Wang

Zhu

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Real-scene Image Super-resolution. Different from the traditional image SR that generally focuses on simple synthesized degradation [32], [33], [34], [35], RealSR [36] needs to handle complicated and rough degradation in real-world scenes [6], [37]. The first attempt is initially to estimate the degradation process for given LR images, and then apply a paired databased model for super-resolution.…”

Section: Related Workmentioning

confidence: 99%

“…θ is the parameters of F (•). To obtain different resolution images in real-scene configuration, I hf and I lf are collected by different optical sensors [6], [7], [8], [9], [10] with various resolution settings, which is different from the traditional image super-resolution paradigm [11], [12], [13], [14], [15], [16] that generates I lf using downsampling techniques. Therefore, compared with the traditional image super-resolution task, RealSR suffers a severer pixel displacement due to the difference between the camera settings to obtain I hf and I lf .…”

Section: Introductionmentioning

confidence: 99%

Criteria Comparative Learning for Real-scene Image Super-Resolution

Shi¹,

Li²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Real-scene image super-resolution aims to restore real-world low-resolution images into their high-quality versions. A typical RealSR framework usually includes the optimization of multiple criteria which are designed for different image properties, by making the implicit assumption that the ground-truth images can provide a good trade-off between different criteria. However, this assumption could be easily violated in practice due to the inherent contrastive relationship between different image properties. Contrastive learning (CL) provides a promising recipe to relieve this problem by learning discriminative features using the triplet contrastive losses. Though CL has achieved significant success in many computer vision tasks, it is non-trivial to introduce CL to RealSR due to the difficulty in defining valid positive image pairs in this case. Inspired by the observation that the contrastive relationship could also exist between the criteria, in this work, we propose a novel training paradigm for RealSR, named Criteria Comparative Learning (Cria-CL), by developing contrastive losses defined on criteria instead of image patches. In addition, a spatial projector is proposed to obtain a good view for Cria-CL in RealSR. Our experiments demonstrate that compared with the typical weighted regression strategy, our method achieves a significant improvement under similar parameter settings.

show abstract

“…Meanwhile, because the estimated image is usually blurred after the decoder process, the generated f (R) is blurry when the edge of f (R) is close to that of I. SR is a process of restoring a high-resolution image from a given low-resolution image [34,35], which is a classic application of computer vision. Specifically, the enhancement of the correlation in the image via adding bicubic interpolation, proposed by Dong et al [36], can be used to improve the resolution of the restored image and improve the visual effect.…”

Section: Network Architecturementioning

confidence: 99%

Transmission Removal from a Single Glass Scene and Its Application in Photographer Identification

Yao

Shi

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

In daily life, when taking photos of scenes containing glass, the images of the dominant transmission layer and the weak reflection layer are often blended, which are difficult to be uncoupled. Meanwhile, because the reflection layer contains sufficient important information about the surrounding scene and the photographer, the problem of recovering the weak reflection layer from the mixture image is of importance in surveillance investigations. However, most of the current studies mainly focus on extracting the transmission layer while often ignoring the merit of the reflection layer. To fill that gap, in this paper, we propose a network framework that aims to accomplish two tasks: (1) for general scenes, we attempt to recover reflection layer images that are as close as possible to the ground truth ones, and (2) for scenes containing portraits, we recover the basic contour information of the reflection layer while improving the defects of dim portraits in the reflection layer. Through analyzing the performance exhibited by different levels of feature maps, we present the first transmission removal network based on an image-to-image translation architecture incorporating residual structures. The quality of generated reflection layer images is improved via tailored content and style constraints. We also use the patch generative adversarial network to increase the discriminator’s ability to perceive the reflection components in the generated images. Meanwhile, the related information such as edge and color distribution of transmission layer in the mixture image is used to assist the overall reflection layer recovery. In the large-scale experiments, our proposed model outperforms reflection removal-based SOTAs by more than 5.356 dB in PSNR, 0.116 in SSIM, and 0.057 in LPIPS.

show abstract

Learning Super-Resolution Reconstruction for High Temporal Resolution Spike Stream

Cited by 17 publications

References 47 publications

Retinomorphic Object Detection in Asynchronous Visual Streams

Retinomorphic Object Detection in Asynchronous Visual Streams

Criteria Comparative Learning for Real-scene Image Super-Resolution

Transmission Removal from a Single Glass Scene and Its Application in Photographer Identification

Contact Info

Product

Resources

About