Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Ignatov, Andrey; Chiang, Cheng-Ming; Kuo, Hsien-Kai; Sycheva, Anastasia; Timofte, Radu; Chen, Min-Hung; Lee, Man-Yu; Xu, Yulong; Tseng, Yu; Xu, Shusong; Guo, Jin; Chen, Chao-Hung; Hsyu, Ming-Chun; Tsai, Wen-Chia; Chen, Chao-Wei; Malivenko, Grigory; Kwon, Minsu; Lee, Myungje; Yoo, Jaeyoon; Kang, Changbeom; Wang, Shinjo; Shaolong, Zheng; Dejun, Hao; Xie, Feng; Zhao, Feng; Ma, Yipeng; Peng, Jingyang; Wang, Tao; Song, Fenglong; Hsu, Chih–Chung; Chen, Kwan-Lin; Wu, Mei-Hsuang; Chudasama, Vishal; Prajapati, Kalpesh; Patel, Heena; Sarvaiya, Anjali; Upla, Kishor P.; Raja, Kiran B.; Ramachandra, Raghavendra; Busch, Christoph; Stoutz, Etienne de

doi:10.1109/cvprw53098.2021.00284

Cited by 30 publications

(44 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Learned Smartphone ISP on Mobile NPUs [22] • Real Image Denoising on Mobile GPUs [21] • Quantized Image Super-Resolution on Edge SoC NPUs [31] • Real-Time Video Super-Resolution on Mobile GPUs [28] • Single-Image Depth Estimation on Mobile Devices…”

Section: Introductionmentioning

confidence: 99%

Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report

Ignatov¹,

Malivenko²,

Plowman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Depth estimation is an important computer vision problem with many practical applications to mobile devices. While many solutions have been proposed for this task, they are usually very computationally expensive and thus are not applicable for on-device inference. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based depth estimation solutions that can demonstrate a nearly realtime performance on smartphones and IoT platforms. For this, the participants were provided with a new large-scale dataset containing RGB-depth image pairs obtained with a dedicated stereo ZED camera producing high-resolution depth maps for objects located at up to 50 meters. The runtime of all models was evaluated on the popular Raspberry Pi 4 platform with a mobile ARM-based Broadcom chipset. The proposed solutions can generate VGA resolution depth maps at up to 10 FPS on the Raspberry Pi 4 while achieving high fidelity results, and are compatible with any Android or Linux-based mobile devices. A detailed description of all models developed in the challenge is provided in this paper.

show abstract

Section: Introductionmentioning

confidence: 99%

Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report

Ignatov¹,

Malivenko²,

Plowman³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…• Learned Smartphone ISP on Mobile NPUs [18] • Real Image Denoising on Mobile GPUs [17] • Quantized Image Super-Resolution on Edge SoC NPUs • Real-Time Video Super-Resolution on Mobile GPUs [25] • Single-Image Depth Estimation on Mobile Devices [21] • Quantized Camera Scene Detection on Smartphones [22] • High Dynamic Range Image Processing on Mobile NPUs…”

Section: Introductionmentioning

confidence: 99%

Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report

Ignatov¹,

Timofte²,

Denna³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Image super-resolution is one of the most popular computer vision problems with many important applications to mobile devices. While many solutions have been proposed for this task, they are usually not optimized even for common smartphone AI hardware, not to mention more constrained smart TV platforms that are often supporting INT8 inference only. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image super-resolution solutions that can demonstrate a real-time performance on mobile or edge NPUs. For this, the participants were provided with the DIV2K dataset and trained quantized models to do an efficient 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated NPU capable of accelerating quantized neural networks. The proposed solutions are fully compatible with all major mobile AI accelerators and are capable of reconstructing Full HD images under 40-60 ms while achieving high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.

show abstract

“…• Learned Smartphone ISP on Mobile NPUs [11] • Real Image Denoising on Mobile GPUs [10] • Quantized Image Super-Resolution on Edge SoC NPUs [19] • Real-Time Video Super-Resolution on Mobile GPUs [16] • Single-Image Depth Estimation on Mobile Devices [13] • Quantized Camera Scene Detection on Smartphones…”

Section: Introductionmentioning

confidence: 99%

Fast and Accurate Quantized Camera Scene Detection on Smartphones, Mobile AI 2021 Challenge: Report

Ignatov¹,

Malivenko²,

Timofte³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Camera scene detection is among the most popular computer vision problem on smartphones. While many custom solutions were developed for this task by phone vendors, none of the designed models were available publicly up until now. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop quantized deep learning-based camera scene classification solutions that can demonstrate a real-time performance on smartphones and IoT platforms. For this, the participants were provided with a large-scale CamSDD dataset consisting of more than 11K images belonging to the 30 most important scene categories. The runtime of all models was evaluated on the popular Apple Bionic A11 platform that can be found in many iOS devices. The proposed solutions are fully compatible with all major mobile AI accelerators and can demonstrate more than 100-200 FPS on the majority of recent smartphone platforms while achieving a top-3 accuracy of more than 98%. A detailed description of all models developed in the challenge is provided in this paper.

show abstract

Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report

Cited by 30 publications

References 27 publications

Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report

Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report

Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report

Fast and Accurate Quantized Camera Scene Detection on Smartphones, Mobile AI 2021 Challenge: Report

Contact Info

Product

Resources

About