NTIRE 2019 Challenge on Image Enhancement: Methods and Results

Ignatov, Andrey; Timofte, Radu; Qu, Xiaobo; Zhou, Xingguang; Liu, Ting; Wan, Pengfei; Zamir, Syed Waqas; Arora, Aditya; Khan, Salman; Khan, Fahad Shahbaz; Shao, Ling; Park, Dong-Won; Chun, Se Young; Michelini, Pablo Navarrete; Liu, Hanwen; Zhu, Dan; Zhong, Zhang; Liu, Xianming; Jiang, Junjun; Zhao, Debin; Haris, Muhammad; Akita, Kazutoshi; Yoshida, Tomoki; Shakhnarovich, Greg; Ukita, Norimichi; Liu, Jie; Jung, Cheolkon; Schettini, Raimondo; Bianco, Simone; Cusano, Claudio; Piccoli, Flavio; Liu, Pengju; Zhang, Kai; Liu, Jingdong; Liu, Jiye; Zhang, Hongzhi; Zuo, Wangmeng; Bow, Nelson Chong Ngee; Wong, Lai-Kuan; See, John; Qin, Jinghui; Huang, Lishan; Shi, Yukai; Wei, Pengxu; Wen, Weisong; Li, Lin; Hui, Zheng; Wang, Xiumei; Gao, Xinbo; Kumari, K.W.S.N.; Anand, Vikas Kumar; Khened, Mahendra; Krishnamurthi, Ganapathy

doi:10.1109/cvprw.2019.00275

“…The collected DPED dataset was later used in many subsequent works [41,9,57,18,35] that have significantly improved the results on this problem. Additionally, in [22] on smartphones, and proposed a number of efficient solutions for this task. It should be mentioned that though the proposed models were showing nice results, they were targeted at refining the images obtained with smartphone ISPs rather than processing RAW camera data.…”

Section: Related Workmentioning

confidence: 99%

Replacing Mobile Camera ISP with a Single Deep Learning Model

Ignatov

¹

,

Gool

²

,

Timofte

³

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP solutions. In this work, we demonstrate that even the most sophisticated ISP pipelines can be replaced with a single end-to-end deep learning model trained without any prior knowledge about the sensor and optics used in a particular device. For this, we present PyNET, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera, making the solution independent of any particular mobile ISP implementation. To validate the proposed approach on the real data, we collected a large-scale dataset consisting of 10 thousand full-resolution RAW-RGB image pairs captured in the wild with the Huawei P20 cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. The experiments demonstrate that the proposed solution can easily get to the level of the embedded P20's ISP pipeline that, unlike our approach, is combining the data from two (RGB + B/W) camera sensors. The dataset, pretrained models and codes used in this paper are available on the project website: https://people.ee.ethz. ch/˜ihnatova/pynet.html

show abstract

“…• Learned End-to-End ISP: [25,29] • Perceptual Image Enhancement: [28,22] • Bokeh Effect Rendering: [21,27] • Image Super-Resolution: [28,41,4,48] Hsyu, Wen-Chia Tsai, Chao-Wei Chen Affiliations: Industrial Technology Research Institute (ITRI), Taiwan…”

Section: Additional Literaturementioning

confidence: 99%

“…The problem of end-to-end mobile photo quality enhancement was first addressed in [16,17], where the authors proposed to enhance all aspects of low-quality smartphone photos by mapping them to superior-quality images obtained with a high-end reflex camera. The collected DPED dataset was later used in many subsequent competitions [28,22] and works [51,40,8,14,13,37] that have significantly improved the results on this problem. While the proposed methods were quite efficient, they worked with the data produced by smartphones' built-in ISPs, thus a significant part of information present in the original sensor data was irrecoverably lost after applying many image processing steps.…”

Section: Introductionmentioning

confidence: 99%

Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Ignatov¹,

Chiang²,

Kuo³

et al. 2021

Preprint

Self Cite

0

View full text Add to dashboard Cite

As the quality of mobile cameras starts to play a crucial role in modern smartphones, more and more attention is now being paid to ISP algorithms used to improve various perceptual aspects of mobile photos. In this Mobile AI challenge, the target was to develop an end-to-end deep learning-based image signal processing (ISP) pipeline that can replace classical hand-crafted ISPs and achieve nearly real-time performance on smartphone NPUs. For this, the participants were provided with a novel learned ISP dataset consisting of RAW-RGB image pairs captured with the Sony IMX586 Quad Bayer mobile sensor and a professional 102megapixel medium format camera. The runtime of all models was evaluated on the MediaTek Dimensity 1000+ platform with a dedicated AI processing unit capable of accelerating both floating-point and quantized neural networks. The proposed solutions are fully compatible with the above NPU and are capable of processing Full HD photos under 60-100 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

show abstract

“…The collected DPED dataset was later used in many subsequent works [41,10,56,19,35] that have significantly improved the results on this problem. Additionally, in [22] the authors examined the possibility of running the resulting image enhancement models directly on smartphones, and proposed a number of efficient solutions for this task. It should be mentioned that though the proposed models were showing nice results, they were targeted at refining the images obtained with smartphone ISPs rather than processing RAW camera data.…”

Section: Related Workmentioning

confidence: 99%

Replacing Mobile Camera ISP with a Single Deep Learning Model

Ignatov¹,

Gool²,

Timofte³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP solutions. In this work, we demonstrate that even the most sophisticated ISP pipelines can be replaced with a single end-to-end deep learning model trained without any prior knowledge about the sensor and optics used in a particular device. For this, we present PyNET, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The model is trained to convert RAW Bayer data obtained directly from mobile camera sensor into photos captured with a professional high-end DSLR camera, making the solution independent of any particular mobile ISP implementation. To validate the proposed approach on the real data, we collected a large-scale dataset consisting of 10 thousand full-resolution RAW-RGB image pairs captured in the wild with the Huawei P20 cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. The experiments demonstrate that the proposed solution can easily get to the level of the embedded P20's ISP pipeline that, unlike our approach, is combining the data from two (RGB + B/W) camera sensors. The dataset, pretrained models and codes used in this paper are available on the project website 1 .

show abstract

NTIRE 2019 Challenge on Image Enhancement: Methods and Results

Cited by 44 publications

References 22 publications

Replacing Mobile Camera ISP with a Single Deep Learning Model

Replacing Mobile Camera ISP with a Single Deep Learning Model

Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Replacing Mobile Camera ISP with a Single Deep Learning Model

Contact Info

Product

Resources

About