NTIRE 2023 Video Colorization Challenge

Kang, Xiaoyang; Lin, Xianhui; Zhang, Kai; Hui, Zheng; Xiang, Wangmeng; He, Jun-Yan; Li, Xiaoming; Ren, Peiran; Xie, Xuansong; Timofte, Radu; Yang, Yixin; Pan, Jinshan; Zheng, Zhong; Qiyan, Peng; Jiangxin, Zhang; Jinhui, Dong; Jinjing, Tang; Chichen, Li; Li, Lin Qipei; Liang, Qirong; Gang, Ruipeng; Liu, Xiaofeng; Feng, Shuang; Liu, Shuai; Wang, Hao; Feng, Chaoyu; Bai, Furui; Zhang, Yuqian; Shao, Guangqi; Wang, Xiaotao; Lei, Lei; Chen, Siqi; Zhang, Yu; Xu, Hanning; Liu, Zheyuan; Zhang, Zhao; Luo, Yan; Zuo, Zhichao

doi:10.1109/cvprw59228.2023.00159

Cited by 24 publications

(2 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These metrics have shown superior correlation with human judgments of VFI quality compared to commonly used quality measurements, PSNR and SSIM (Wang et al 2004). We also evaluate FID (Heusel et al 2017) which measures the similarity between the distributions of interpolated and ground-truth frames; this was previously used as a perceptual metric for video compression (Yang, Timofte, and Van Gool 2022), enhancement (Yang 2021) and colorization (Kang et al 2023).…”

Section: Methodsmentioning

confidence: 99%

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Danier,

Zhang,

Bull

2024

AAAI

View full text Add to dashboard Cite

Existing works on video frame interpolation (VFI) mostly employ deep neural networks that are trained by minimizing the L1, L2, or deep feature space distance (e.g. VGG loss) between their outputs and ground-truth frames. However, recent works have shown that these metrics are poor indicators of perceptual VFI quality. Towards developing perceptually-oriented VFI methods, in this work we propose latent diffusion model-based VFI, LDMVFI. This approaches the VFI problem from a generative perspective by formulating it as a conditional generation problem. As the first effort to address VFI using latent diffusion models, we rigorously benchmark our method on common test sets used in the existing VFI literature. Our quantitative experiments and user study indicate that LDMVFI is able to interpolate video content with favorable perceptual quality compared to the state of the art, even in the high-resolution regime. Our code is available at https://github.com/danier97/LDMVFI.

show abstract

Section: Methodsmentioning

confidence: 99%

LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Danier,

Zhang,

Bull

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…The inherent limitation arises from the absence of color information in both low dynamic range (LDR) images and event streams, as shown in Figure 14. A prospective solution to this issue could involve the integration of image colorization techniques [69,70]…”

Section: Limitations and Future Workmentioning

confidence: 99%

ERS-HDRI: Event-Based Remote Sensing HDR Imaging

Li,

Cheng,

Zeng

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

High dynamic range imaging (HDRI) is an essential task in remote sensing, enhancing low dynamic range (LDR) remote sensing images and benefiting downstream tasks, such as object detection and image segmentation. However, conventional frame-based HDRI methods may encounter challenges in real-world scenarios due to the limited information inherent in a single image captured by conventional cameras. In this paper, an event-based remote sensing HDR imaging framework is proposed to address this problem, denoted as ERS-HDRI, which reconstructs the remote sensing HDR image from a single-exposure LDR image and its concurrent event streams. The proposed ERS-HDRI leverages a coarse-to-fine framework, incorporating the event-based dynamic range enhancement (E-DRE) network and the gradient-enhanced HDR reconstruction (G-HDRR) network. Specifically, to efficiently achieve dynamic range fusion from different domains, the E-DRE network is designed to extract the dynamic range features from LDR frames and events and perform intra- and cross-attention operations to adaptively fuse multi-modal data. A denoise network and a dense feature fusion network are then employed for the generation of the coarse, clean HDR image. Then, the G-HDRR network, with its gradient enhancement module and multiscale fusion module, performs structure enforcement on the coarse HDR image and generates a fine informative HDR image. In addition, this work introduces a specialized hybrid imaging system and a novel, real-world event-based remote sensing HDRI dataset that contains aligned remote sensing LDR images, remote sensing HDR images, and concurrent event streams for evaluation. Comprehensive experiments have demonstrated the effectiveness of the proposed method. Specifically, it improves state-of-the-art PSNR by about 30% and the SSIM score by about 9% on the real-world dataset.

show abstract