Robust super-resolution by fusion of interpolated frames for color and grayscale images

Abstract

Multi-frame super-resolution (SR) processing seeks to overcome undersampling issues that can lead to undesirable aliasing artifacts in imaging systems. A key factor in effective multi-frame SR is accurate subpixel inter-frame registration. Accurate registration is more difficult when frame-to-frame motion does not contain simple global translation and includes locally moving scene objects. SR processing is further complicated when the camera captures full color by using a Bayer color filter array (CFA). Various aspects of these SR challenges have been previously investigated. Fast SR algorithms tend to have difficulty accommodating complex motion and CFA sensors. Furthermore, methods that can tolerate these complexities tend to be iterative in nature and may not be amenable to real-time processing. In this paper, we present a new fast approach for performing SR in the presence of these challenging imaging conditions. We refer to the new approach as Fusion of Interpolated Frames (FIF) SR. The FIF SR method decouples the demosaicing, interpolation, and restoration steps to simplify the algorithm. Frames are first individually demosaiced and interpolated to the desired resolution. Next, FIF uses a novel weighted sum of the interpolated frames to fuse them into an improved resolution estimate. Finally, restoration is applied to improve any degrading camera effects. The proposed FIF approach has a lower computational complexity than many iterative methods, making it a candidate for real-time implementation. We provide a detailed description of the FIF SR method and show experimental results using synthetic and real datasets in both constrained and complex imaging scenarios. Experiments include airborne grayscale imagery and Bayer CFA image sets with affine background motion plus local motion.

Fusion of interpolated frames superresolution in the presence of atmospheric optical turbulence

“…A small β would be expected to give the best SR results with minimal interpolation error, provided a sufficient diversity of samples is available with a high signal-to-noise ratio. 17 The tunability provided by the parameter β sets it apart from most other nonuniform interpolation SR methods 2 and makes it well suited to perform SR in the presence of turbulence. For example, it is interesting to compare the FIF SR approach of fusing interpolated frames with methods that use binning.…”

“…This result is based on the fraction of frames expected to fall into each bin under a uniform subpixel displacement assumption. It should be noted that the original FIF SR paper 17 considers other weighting components, in addition to that in Eq. (2).…”

“…A framework that is well suited to balance the factors described above is the fusion of interpolated frames (FIF) SR method, recently proposed by two of the current authors. 17 The FIF SR method is a multiframe SR algorithm that fuses interpolated versions of each low resolution (LR) input frame. The fusion weights are based on the subpixel alignment for each interpolated pixel, any color information that may be available, and estimated local scene motion activity.…”

“…5. Figure 1 shows a block diagram summarizing the FIF SR method, originally from Karch and Hardie, 17 and adapted here for joint SR and TM. In our implementation, shortexposure LR observed frames are registered using one of a variety of registration methods appropriate to the application.…”

Novel Graph Cuts Method for Multi-Frame Super-Resolution

“…They showed that HR images recovered after minimizing an energy function which includes this notion of zone of influence are of good quality, especially when the HR image is larger than the LR images. Jamalabadi and Ahmadyfard [10] recently proposed a method which gives different weights to the LR images while Karch and Hardie [16] pushed the idea further by using 3 types of weights: an interpolation weight, a local motion weight, and channel cross-correlation weight.…”