Event-Based Fusion for Motion Deblurring with Cross-modal Attention

Sun, Lei; Sakaridis, Christos; Liang, Jingyun; Jiang, Qi; Yang, Kailun; Sun, Peng; Ye, Yaozu; Wang, Kaiwei; Gool, Luc Van

doi:10.1007/978-3-031-19797-0_24

Cited by 41 publications

(20 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jiang et al [14] used convolutional models and mined the motion information and edge information to assist deblurring. Sun et al [35] proposed a multihead attention mechanism for fusing information from both modalities, and designed an event representation specifically for the event-based image deblurring task. Kim et al [15] further extended the task to images with unknown exposure time by activating the events that are most related to the blurry image.…”

Section: Related Workmentioning

confidence: 99%

“…3 shows the detailed architecture of the proposed EGACA. We simplify the multi-head channel attention of EFNet [35] to channel attention from SENet [10]. Two Channel Squeeze (CS) blocks extract channel weights from the current events, and two weights multiply event features and image features for self-attention and event-guided attention to image features, respectively.…”

Section: Event-guided Adaptive Channel Attentionmentioning

confidence: 99%

“…For event-based low-level tasks, such as event-based image deblurring and event-based frame interpolation, most works evaluate their models on datasets originally designed for image-only methods which only have synthetic events. This is because (1) event cameras are not easy to get yet, (2) most event cameras are of low resolution and monochrome [14,15,35], and (3) high-resolution chromatic datasets [38,39] are not publicly available. To fill this gap, we record a high-quality chromatic event-image dataset for training, fine-tuning and evaluating event-based methods for frame interpolation and deblurring.…”

Section: Highrev Datasetmentioning

confidence: 99%

“…We use the original GoPro dataset with synthesized blurry images (averaged from 7 or 11 sharp frames). To test on a real-world dataset, we also fine-tune and evaluate methods on the REBlur dataset [35]. We only use a single image and its corresponding events in the event branch as input, for a fair comparison.…”

Section: Tasks and Datasetsmentioning

confidence: 99%

“…Since most event-based methods do not have public implementations, we use "E2VID+" by adding an extra encoder for images and introduce images as extra inputs for the event-based image reconstruction method E2VID [26]. As a two-stage method, we use EFNet+IFRNet by combining a state-of-the-art event-based image deblurring method [35] with an image-only frame interpolation method [17]. For a fair comparison, IFRNet is also fed with event voxels from two directions as inputs.…”

Section: Blurry Frame Interpolationmentioning

confidence: 99%

See 4 more Smart Citations

Event-Based Frame Interpolation with Ad-hoc Deblurring

Sun¹,

Sakaridis²,

Liang³

et al. 2023

Preprint

View full text Add to dashboard Cite

The performance of video frame interpolation is inherently correlated with the ability to handle motion in the input scene. Even though previous works recognize the utility of asynchronous event information for this task, they ignore the fact that motion may or may not result in blur in the input video to be interpolated, depending on the length of the exposure time of the frames and the speed of the motion, and assume either that the input video is sharp, restricting themselves to frame interpolation, or that it is blurry, including an explicit, separate deblurring stage before interpolation in their pipeline. We instead propose a general method for event-based frame interpolation that performs deblurring ad-hoc and thus works both on sharp and blurry input videos. Our model consists in a bidirectional recurrent network that naturally incorporates the temporal dimension of interpolation and fuses information from the input frames and the events adaptively based on their temporal proximity. In addition, we introduce a novel real-world high-resolution dataset with events and color videos named HighREV, which provides a challenging evaluation setting for the examined task. Extensive experiments on the standard GoPro benchmark and on our dataset show that our network consistently outperforms previous state-of-the-art methods on frame interpolation, single image deblurring and the joint task of interpolation and deblurring. Our code and dataset will be made publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Event-guided Adaptive Channel Attentionmentioning

confidence: 99%