Deep Residual Network for Joint Demosaicing and Super-Resolution

Zhou, Ruofan; Achanta, Radhakrishna; Süsstrunk, Sabine

doi:10.2352/issn.2169-2629.2018.26.75

Cited by 38 publications

(54 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where N (x) = (A − I)x + n is the image-dependent noise function. Instead of solving A and n, we decide to estimate N (x) directly since it is more solvable under the deep learning framework [13,14,15,16,17]. Essentially, by estimating N (x) andx, we aim to peel off the spoof noise and reconstruct the original live face.…”

Section: Introductionmentioning

confidence: 99%

Face De-spoofing: Anti-spoofing via Noise Modeling

Jourabloo

Liu

2018

Lecture Notes in Computer Science

233

162

View full text Add to dashboard Cite

Many prior face anti-spoofing works develop discriminative models for recognizing the subtle differences between live and spoof faces. Those approaches often regard the image as an indivisible unit, and process it holistically, without explicit modeling of the spoofing process. In this work, motivated by the noise modeling and denoising algorithms, we identify a new problem of face despoofing, for the purpose of anti-spoofing: inversely decomposing a spoof face into a spoof noise and a live face, and then utilizing the spoof noise for classification. A CNN architecture with proper constraints and supervisions is proposed to overcome the problem of having no ground truth for the decomposition. We evaluate the proposed method on multiple face anti-spoofing databases. The results show promising improvements due to our spoof noise modeling. Moreover, the estimated spoof noise provides a visualization which helps to understand the added spoof noise by each spoof medium.

show abstract

Section: Introductionmentioning

confidence: 99%

Face De-spoofing: Anti-spoofing via Noise Modeling

Jourabloo

Liu

2018

Lecture Notes in Computer Science

233

162

View full text Add to dashboard Cite

show abstract

“…Many existing methods for this problem estimate a high-resolution color image with multiple low-resolution frames [7,33]. More closely related to our task, Zhou et al [39] propose a deep residual network for single image super-resolution with mosaiced images. However, this model is trained on gammacorrected image pairs which may not work well for real linear data.…”

Section: Super-resolutionmentioning

confidence: 99%

“…However, as introduced in Section 1, the low-resolution images generated in this way does not resemble real captured images and will be less effective for training real scene super-resolution models. Moreover, we need to generate low-resolution raw data as well for training the proposed dual CNN, which is often approached by directly mosaicing the low-resolution color images [9,39]. This strategy ignores the fact that the color images have been processed by nonlinear operations of the ISP system while the raw data should be from linear color measurements of the pixels.…”

Section: Training Datamentioning

confidence: 99%

“…Thus, the linear degradations in the imaging process, including blur and noise, are nonlinear in the processed RGB space, which brings more difficulties in image restoration [24]. (iii) The demosaicing step in the ISP is highly related to super-resolution, because these two problems both refer to the resolution limitations of cameras [39]. Therefore, to solve the super-resolution problem with preprocessed images is sub-optimal and could be inferior to a single unified model solving both problems simultaneously.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Real Scene Super-Resolution With Raw Images

Xu¹,

Ma²,

Sun³

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

109

View full text Add to dashboard Cite

Figure 1. Existing super-resolution method (RDN [38]) does not perform well for real captured images as shown in (c). We can obtain sharper results (d) by re-training existing model [38] with the data generated by our method. Furthermore, we recover more structures and details (e) by exploiting the radiance information recorded in raw images. The two input images (a) are captured by Leica SL Typ-601 and iPhone 6s respectively, and both cameras are not seen by the models during training. Results best viewed electronically with zoom. AbstractMost existing super-resolution methods do not perform well in real scenarios due to lack of realistic training data and information loss of the model input. To solve the first problem, we propose a new pipeline to generate realistic training data by simulating the imaging process of digital cameras. And to remedy the information loss of the input, we develop a dual convolutional neural network to exploit the originally captured radiance information in raw images. In addition, we propose to learn a spatially-variant color transformation which helps more effective color corrections. Extensive experiments demonstrate that superresolution with raw data helps recover fine details and clear structures, and more importantly, the proposed network and data generation pipeline achieve superior results for single image super-resolution in real scenarios.

show abstract

“…[2017] investigated various loss functions for image restoration. Other methods perform joint demosaicing and denoising include methods proposed by Hirakawa and Parks [2006], Khashabi et al [2014], Klatzer et al [2016], , Condat [2010], Menon and Calvagno [2009], Goossens et al [2013], , Hirakawa [2008], Zhou et al [2018], Fang et al [2012], Klatzer et al [2016], Condat and Mosaddegh [2012], and Henz et al [2018].…”

Section: Related Workmentioning

confidence: 99%

Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing

Ratnasingam

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

a) Our CNN (b) Ground truth Figure 1: Our convolutional neural network performs the entire processing involved in an image signal processing (ISP) pipeline including denoising, white balancing, exposure correction, demosaicing, color transform, and gamma encoding. Results for our method and ground truth image are shown: (a) our CNN-based ISP and (b) ground truth image. In each image, a zoomed in view of the white rectangular region is displayed in the top left hand corner (inside the black rectangle). It can be seen that the CNN output looks almost identical to the ground truth image. ABSTRACTA conventional camera performs various signal processing steps sequentially to reconstruct an image from a raw Bayer image. When performing these processing in multiple stages the residual error from each stage accumulates in the image and degrades the quality of the final reconstructed image. In this paper, we present a fully convolutional neural network (CNN) to perform defect pixel correction, denoising, white balancing, exposure correction, demosaicing, color transform, and gamma encoding. To our knowledge, this is the first CNN trained end-to-end to perform the entire image signal processing pipeline in a camera. The neural network was trained using a large image database of raw Bayer images. Through extensive experiments, we show that the proposed CNN based image signal processing system performs better than the conventional signal processing pipelines that perform the processing sequentially.

show abstract

Deep Residual Network for Joint Demosaicing and Super-Resolution

Cited by 38 publications

References 20 publications

Face De-spoofing: Anti-spoofing via Noise Modeling

Face De-spoofing: Anti-spoofing via Noise Modeling

Towards Real Scene Super-Resolution With Raw Images

Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing

Contact Info

Product

Resources

About