Hyperparameter optimization in black-box image processing using differentiable proxies

Abstract

Nearly every commodity imaging system we directly interact with, or indirectly rely on, leverages power efficient, application-adjustable black-box hardware image signal processing (ISPs) units, running either in dedicated hardware blocks, or as proprietary software modules on programmable hardware. The configuration parameters of these black-box ISPs often have complex interactions with the output image, and must be adjusted prior to deployment according to application-specific quality and performance metrics. Today, this search is commonly performed manually by "golden eye" experts or algorithm developers leveraging domain expertise. We present a fully automatic system to optimize the parameters of black-box hardware and software image processing pipelines according to any arbitrary (i.e., application-specific) metric. We leverage a differentiable mapping between the configuration space and evaluation metrics, parameterized by a convolutional neural network that we train in an end-to-end fashion with imaging hardware in-the-loop. Unlike prior art, our differentiable proxies allow for high-dimension parameter search with stochastic first-order optimizers, without explicitly modeling any lower-level image processing transformations. As such, we can efficiently optimize black-box image processing pipelines for a variety of imaging applications, reducing application-specific configuration times from months to hours. Our optimization method is fully automatic, even with black-box hardware in the loop. We validate our method on experimental data for real-time display applications, object detection, and extreme low-light imaging. The proposed approach outperforms manual search qualitatively and quantitatively for all domain-specific applications tested. When applied to traditional denoisers, we demonstrate that---just by changing hyperparameters---traditional algorithms can outperform recent deep learning methods by a substantial margin on recent benchmarks.

Programming with neural surrogates of programs

“…With surrogate optimization, programmers develop a surrogate of a program, optimize input parameters of that surrogate, then finally plug the optimized input parameters back into the original program. The key benefit of this approach is that surrogate optimization can optimize inputs faster than optimizing inputs directly against the program, due to the potential for faster execution speed of the surrogate and the potential for the surrogate to be differentiable even when the original program is not (allowing for optimizing inputs with gradient descent) [71,77,87].…”

“…Typical examples of surrogates include neural networks [27,71], Gaussian processes [3,69], linear models [22,25], and random forests [38,62]. Of these model architectures, neural surrogates have emerged as a popular design for surrogates in the literature [23,42,71,77,84,87] because for many tasks neural networks are state-of-the-art models that lead to high accuracy [20,47].…”

“…Programmers use surrogates for a variety of tasks including accelerating computational kernels in numerical programs [23], replacing physical simulators with more accurate versions [84], and tuning parameters of complex simulators [71,87]. Compared to standard development workflows, programming with surrogates requires lower development costs [45,49,71,77,87] and results in programs with lower execution cost [23,57,60,67] or higher result quality [48,71,84,87]. However, the approaches in the literature for both applying and developing surrogates are disparate, with no unifying taxonomy or development methodology.…”

Efficient Hyperparameter Optimization for Physics-based Character Animation

“…We also adopt a Bayesian optimization framework, but for hyperparameter optimization. To the best of our knowledge, the only previous work on hyperparameter optimization in graphics is [Tseng et al 2019], where parameters of image processing hardware, such as threshold values in the denoising module, are optimized with a non-Bayesian approach. Hereafter we will focus on hyperparameter optimization literature from machine learning and robotics.…”

Real-time Defogging of Single Image of IoTs-based Surveillance Video Based on MAP

“…Step 5: the atmospheric light intensity G is used to calculate and optimize the transmission image, then optimize the field depth of the IoTs surveillance video image, and in actual application, even in sunny and foggy weather, the IoTs surveillance video will also be interfered by atmospheric particles. If all the fog is removed, it will reduce the authenticity of the image, so use the adjustment coefficient σ to retain a small amount of fog and optimize the image authenticity [24]. Therefore the projection is set to the following relationship.…”