PyNET-CA: Enhanced PyNET with Channel Attention for End-to-End Mobile Image Signal Processing

Kim, Byung-Hoon; Song, Joonyoung; Ye, Jong Chul; Baek, Jehyun

doi:10.1007/978-3-030-67070-2_12

Cited by 13 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PyNET [1] significantly enhanced performance by using an inverted pyramidal structure with various convolution filters. PyNET-CA [30] incorporated a channel attention mechanism into PyNET, further improving the reconstruction quality.…”

Section: Related Work a Deep Learning Based Demosaicingmentioning

confidence: 99%

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

Cho

Lee

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Deep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color filter arrays (CFA) such as Quad Bayer, Nona Bayer, and Q×Q Bayer to enhance image quality, yet most existing deep learning-based ISP (or demosaicing) models focus primarily on standard Bayer CFAs. In this study, we present PyNET-Q×Q, a lightweight demosaicing model specifically designed for Q×Q Bayer CFA patterns, which is derived from the original PyNET. We also propose a knowledge distillation method called progressive distillation to train the reduced network more effectively. Consequently, PyNET-Q×Q contains less than 2.5% of the parameters of the original PyNET while preserving its performance. Experiments using Q×Q images captured by a prototype Q×Q camera sensor show that PyNET-Q×Q outperforms existing conventional algorithms in terms of texture and edge reconstruction, despite its significantly reduced parameter count. Code and partial datasets can be found at https://github.com/Minhyeok01/PyNET-QxQ.INDEX TERMS Bayer filter, color filter array (CFA), demosaicing, image signal processor (ISP), knowledge distillation, non-Bayer CFA, Q×Q Bayer CFA.

show abstract

Section: Related Work a Deep Learning Based Demosaicingmentioning

confidence: 99%

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

Cho

Lee

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Skyb presented a PyNet-CA model [25] (Fig. 8) that adds several enhancements on top of the standard PyNET [24] architecture.…”

Section: Skybmentioning

confidence: 99%

AIM 2020 Challenge on Learned Image Signal Processing Pipeline

Ignatov

Timofte

Zhang

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper reviews the second AIM learned ISP challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world RAW-to-RGB mapping problem, where to goal was to map the original low-quality RAW images captured by the Huawei P20 device to the same photos obtained with the Canon 5D DSLR camera. The considered task embraced a number of complex computer vision subtasks, such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The target metric used in this challenge combined fidelity scores (PSNR and SSIM) with solutions' perceptual results measured in a user study. The proposed solutions significantly improved the baseline results, defining the state-of-the-art for practical image signal processing pipeline modeling. * A. Ignatov and R. Timofte ({andrey,radu.timofte}@vision.ee.ethz.ch, ETH Zurich) are the challenge organizers, while the other authors participated in the challenge. The Appendix A contains the authors' teams and affiliations. AIM 2020 webpage: https://data.vision.ee.ethz.ch/cvl/aim20/

show abstract

“…For this, a Zurich RAW to RGB dataset containing RAW-RGB image pairs from a mobile camera sensor and a high-end DSLR camera was collected. The proposed learned ISP reached the quality level of commercial ISP system of the Huawei P20 camera phone, and these results were further improved in [37,7,60,43,33]. In this challenge, we use a more advanced FujiFlim UltraISP dataset [27,23] and additional efficiency-related constraints on the developed solutions.…”

Section: Introductionmentioning

confidence: 99%

Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

Ignatov¹,

Timofte²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The role of mobile cameras increased dramatically over the past few years, leading to more and more research in automatic image quality enhancement and RAW photo processing. In this Mobile AI challenge, the target was to develop an efficient end-to-end AI-based image signal processing (ISP) pipeline replacing the standard mobile ISPs that can run on modern smartphone GPUs using TensorFlow Lite. The participants were provided with a large-scale Fujifilm UltraISP dataset consisting of thousands of paired photos captured with a normal mobile camera sensor and a professional 102MP medium-format FujiFilm GFX100 camera. The runtime of the resulting models was evaluated on the Snapdragon's 8 Gen 1 GPU that provides excellent acceleration results for the majority of common deep learning ops. The proposed solutions are compatible with all recent mobile GPUs, being able to process Full HD photos in less than 20-50 milliseconds while achieving high fidelity results. A detailed description of all models developed in this challenge is provided in this paper.

show abstract

PyNET-CA: Enhanced PyNET with Channel Attention for End-to-End Mobile Image Signal Processing

Cited by 13 publications

References 23 publications

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

AIM 2020 Challenge on Learned Image Signal Processing Pipeline

Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

Contact Info

Product

Resources

About