Deep Demosaicing for Edge Implementation

Abstract

Most digital cameras use sensors coated with a Color Filter Array (CFA) to capture channel components at every pixel location, resulting in a mosaic image that does not contain pixel values in all channels. Current research on reconstructing these missing channels, also known as demosaicing, introduces many artifacts, such as zipper effect and false color. Many deep learning demosaicing techniques outperform other classical techniques in reducing the impact of artifacts. However, most of these models tend to be over-parametrized. Consequently, edge implementation of the state-of-the-art deep learning-based demosaicing algorithms on low-end edge devices is a major challenge. We provide an exhaustive search of deep neural network architectures and obtain a Pareto front of Color Peak Signal to Noise Ratio (CPSNR) as the performance criterion versus the number of parameters as the model complexity that outperforms the state-of-the-art. Architectures on the Pareto front can then be used to choose the best architecture for a variety of resource constraints. Simple architecture search methods such as exhaustive search and grid search requires some conditions of the loss function to converge to the optimum. We clarify these conditions in a brief theoretical study.

Make the Most Out of Your Net: Alternating Between Canonical and Hard Datasets for Improved Image Demosaicing

“…Initially, the method identifies difficult patch samples in the original training dataset and divides them into sub-categories.Then, the model is trained in an approach we call cyclic training, using the sub-categories to guide the learning process, in a non-standard scheme, alternating between the entire dataset and the generated sub-categories. In addition, there is a recent trend of developing low-capacity models (less than 50k parameters) for edge devices to perform image demosaicing [19,25,34]. We show that our method is able to effectively utilize the model's capacity and surpass recent relevant works across all benchmarks using fewer number of parameters, showcasing a more efficient solution for low-capacity devices.…”

“…We compare our method with recent studies that focus on achieving high performance for image demosaicing using low capacity models. In [25] 1)-( 3) was similar, with an initial learning rate of 5 × 10 −4 , that was reduced by half after every 100K iterations. The models were trained for a total of 2M iterations.…”

“…The task of image demosaicing is a crucial step in digital photography and refers to the process of reconstructing a full-resolution RGB color image from incomplete data obtained from the use of a color filter array (CFA), such as the Bayer pattern of GRBG. In digital cameras, CFA samples only a fraction of the image information, which Figure 1: PSNR results of our approach (pink stars) applied on various architectures and compared to leading previous works [7,25,36,39,42,38] over Kodak [15] dataset. Using the suggested training method we achieve better results than all other networks of the same amount of parameters.…”

HISP: Heterogeneous Image Signal Processor Pipeline Combining Traditional and Deep Learning Algorithms Implemented on FPGA

“…There is no doubt that the emergence of a large number of neural networks for denoising and enhancing images is essentially due to the extremely significant advantages of deep learning in these two types of tasks in image processing [44][45][46].…”