Towards Interpretable Video Super-Resolution via Alternating Optimization

Cao, Jiezhang; Liang, Jingyun; Zhang, Kai; Yang, Yi; Wang, Qin; Zhang, Yulun; Tang, Hao; Gool, Luc Van

doi:10.1007/978-3-031-19797-0_23

Cited by 8 publications

(2 citation statements)

References 61 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…age restoration [92], and image/video super-resolution [93], [94]. Moreover, we follow OASIS [53] and use LPIPS [95] to evaluate the variation in the multi-model image synthesis on the ADE20K dataset.…”

Section: Resultsmentioning

confidence: 99%

Dual Attention GANs for Semantic Image Synthesis

Tang

Bai

Sebe

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

We propose a novel edge guided generative adversarial network with contrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling "local" semantic information from a single input semantic layout. However, they ignore "global" semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales. Experiments on three challenging datasets show that our methods achieve significantly better results than state-of-the-art approaches. The source code is available at https://github.com/Ha0Tang/ECGAN.

show abstract

Section: Resultsmentioning

confidence: 99%

Dual Attention GANs for Semantic Image Synthesis

Tang

Bai

Sebe

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…CNNs has gained widespread popularity in various lowlevel image processing tasks, including image deblurring [15,32,50], denoising [14,52], and super-resolution [9,47]. Although CNNs have been effectively employed in demosaicing [23,35,48], their application has been predominantly limited to the Bayer pattern, which has a predominant green band.…”

Section: Hsi Demosaicingmentioning

confidence: 99%

MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing

Zeng¹,

Feng²,

Shaoguang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Hyperspectral imaging systems that use multispectral filter arrays (MSFA) capture only one spectral component in each pixel. Hyperspectral demosaicing is used to recover the non-measured components. While deep learning methods have shown promise in this area, they still suffer from several challenges, including limited modeling of non-local dependencies, lack of consideration of the periodic MSFA pattern that could be linked to periodic artifacts, and difficulty in recovering high-frequency details. To address these challenges, this paper proposes a novel demosaicing framework, the MSFA-frequency-aware Transformer network (FDM-Net). FDM-Net integrates a novel MSFA-frequency-aware multi-head self-attention mechanism (MaFormer) and a filter-based Fourier zero-padding method to reconstruct high pass components with greater difficulty and low pass components with relative ease, separately. The advantage of Maformer is that it can leverage the MSFA information and non-local dependencies present in the data. Additionally, we introduce a joint spatial and frequency loss to transfer MSFA information and enhance training on frequency components that are hard to recover. Our experimental results demonstrate that FDM-Net outperforms state-of-the-art methods with 6dB PSNR, and reconstructs high-fidelity details successfully.

show abstract