Optical Aberrations Correction in Postprocessing Using Imaging Simulation

Abstract

As the popularity of mobile photography continues to grow, considerable effort is being invested in the reconstruction of degraded images. Due to the spatial variation in optical aberrations, which cannot be avoided during the lens design process, recent commercial cameras have shifted some of these correction tasks from optical design to postprocessing systems. However, without engaging with the optical parameters, these systems only achieve limited correction for aberrations. In this work, we propose a practical method for recovering the degradation caused by optical aberrations. Specifically, we establish an imaging simulation system based on our proposed optical point spread function model. Given the optical parameters of the camera, it generates the imaging results of these specific devices. To perform the restoration, we design a spatial-adaptive network model on synthetic data pairs generated by the imaging simulation system, eliminating the overhead of capturing training data by a large amount of shooting and registration. Moreover, we comprehensively evaluate the proposed method in simulations and experimentally with a customized digital-single-lens-reflex camera lens and HUAWEI HONOR 20, respectively. The experiments demonstrate that our solution successfully removes spatially variant blur and color dispersion. When compared with the state-of-the-art deblur methods, the proposed approach achieves better results with a lower computational overhead. Moreover, the reconstruction technique does not introduce artificial texture and is convenient to transfer to current commercial cameras. Project Page: https://github.com/TanGeeGo/ImagingSimulation .

Annular Computational Imaging: Capture Clear Panoramic Images through Simple Lens

“…As data-driven deep learning methods are extensively applied in image restoration [37]- [39], more aberration correction works are equipped with powerful CNN. These approaches often utilize networks with encoder-decoder structure [40] combined with efficient blocks for complex spatial distribution of aberrations [19], [20], [22], [23], [41], obtaining more efficient and robust results than traditional methods. Other networks for image restoration with spatially diverse degradation are also fit for aberration correction which fuse multi-scale information [38], [42], and use deformable convolution [43], [44] or dynamic convolution [45].…”

“…Consequently, imaging simulation [20], [22], [41] becomes a preferable choice by generating synthetic aberration datasets. The key point is to calculate the PSFs under optical aberration, which are often simulated by raytracing or wave-based methods.…”

“…In the last decade, the great success of Convolutional Neural Networks (CNNs) promotes the advance of Computational Imaging (CI) methods, in which a main group of efforts attempts to restore the aberration images from simple, imperfect optical lens [19]- [22]. Apart from network design, the key to applying CNN is to prepare datasets for network training, which is often realized by live shooting [19], [23] and optical simulation based on raytracing [22], random Zernike coefficients [20] or optical design software [24]. However, most of the popular CI approaches [25]- [28] only deal with the pinhole camera lenses with limited FoV, whose distributions of degradation and FoV are different from those of PAL.…”

“…On the other hand, deep learning based aberration correction [20], [22] often regards the problem as image restoration which discards the physical model of imaging. To integrate physical priors into restoration, we adopt the idea of the Physics-Informed Neural Network (PINN) [29] and propose the Physics Informed Image Restoration Network (PI 2 RNet).…”

“…The followed fine restoration is also improved by degradation characteristics from physics-informed engine, specially designed convolution kernel and global context components. Last but not least, although synthetic datasets [22], [24] are easier to be mass produced and transferred to different lenses compared to manual shooting [19], [23], the syntheticto-real gap is non-negligible considering that the manufactured lenses often deviate from the theoretical results. In our ACI framework, optical simulation is considered as data augmentation, where the random distributions of PSFs contribute to addressing errors introduced by manufacture and assembly, facilitating loose-tolerance requirements of our framework for lens design.…”