FIFNET: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames

Elwarfalli, Hamed; Hardie, Russell C.

doi:10.1016/j.cviu.2020.103097

Cited by 10 publications

(14 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The CNN architecture we use here is based on the FIFNet. 38 The FIFNet has a lightweight architecture with relatively few layers. This allows us to train the network from scratch, without the need for transfer learning, in ∼100 min.…”

Section: Discussionmentioning

confidence: 99%

“…The advantage of this is that it allows for a smaller network and avoids extrapolation errors on the patch borders. 38 CNN updates during training are done using stochastic gradient descent with momentum optimization. Sixty four different random patches are extracted from each training image.…”

Section: Block Matching and Convolutional Neural Networkmentioning

confidence: 99%

“…Description of the FIFNet layers38 used for the proposed BM-CNN TM method. The spatial dimensions of the activations are for training using 61 × 61 image patches.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Deep learning for anisoplanatic optical turbulence mitigation in long-range imaging

et al. 2021

Self Cite

View full text Add to dashboard Cite

We present a deep learning approach for restoring images degraded by atmospheric optical turbulence. We consider the case of terrestrial imaging over long ranges with a wide fieldof-view. This produces an anisoplanatic imaging scenario where turbulence warping and blurring vary spatially across the image. The proposed turbulence mitigation (TM) method assumes that a sequence of short-exposure images is acquired. A block matching (BM) registration algorithm is applied to the observed frames for dewarping, and the resulting images are averaged. A convolutional neural network (CNN) is then employed to perform spatially adaptive restoration. We refer to the proposed TM algorithm as the block matching and CNN (BM-CNN) method. Training the CNN is accomplished using simulated data from a fast turbulence simulation tool capable of producing a large amount of degraded imagery from declared truth images rapidly. Testing is done using independent data simulated with a different well-validated numerical wave-propagation simulator. Our proposed BM-CNN TM method is evaluated in a number of experiments using quantitative metrics. The quantitative analysis is made possible by virtue of having truth imagery from the simulations. A number of restored images are provided for subjective evaluation. We demonstrate that the BM-CNN TM method outperforms the benchmark methods in the scenarios tested. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Block Matching and Convolutional Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning for anisoplanatic optical turbulence mitigation in long-range imaging

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…To enrich the RCAN input, we also supply subpixel interpolation distances for each pixel and feed into the RCAN architecture through a separate input head. This modification is inspired by the FIFNET architecture for multi-image SR. 33 Since paired hexagonally sampled LR images and rectangularly sampled HR images are not readily available, we develop an observation model-based approach for generating realistic, full frame RGB, synthetic data. Our observation model effectively constitutes a "forward model" for the inverse problem we wish to solve.…”

Section: Introductionmentioning

confidence: 99%

“…Our observation model effectively constitutes a "forward model" for the inverse problem we wish to solve. The observation model leverages a camera-specific optical transfer function (OTF) that models diffraction and detector integration (based on either hexagonal or rectangular detectors) for generating our data 33,34 as well as a resampling step. We apply the observation model to the DIV2K SISR dataset 35 to generate data to train our models so as to be ready to process real full frame RGB color camera outputs.…”

Section: Introductionmentioning

confidence: 99%

Resampling and super-resolution of hexagonally sampled images using deep learning

Flaute,

Hardie,

Elwarfalli

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Super-resolution (SR) aims to increase the resolution of imagery. Applications include security, medical imaging, and object recognition. We propose a deep learning-based SR system that takes a hexagonally sampled lowresolution image as an input and generates a rectangularly sampled SR image as an output. For training and testing, we use a realistic observation model that includes optical degradation from diffraction and sensor degradation from detector integration. Our SR approach first uses non-uniform interpolation to partially upsample the observed hexagonal imagery and convert it to a rectangular grid. We then leverage a state-of-the-art convolutional neural network (CNN) architecture designed for SR known as Residual Channel Attention Network (RCAN). In particular, we use RCAN to further upsample and restore the imagery to produce the final SR image estimate. We demonstrate that this system is superior to applying RCAN directly to rectangularly sampled LR imagery with equivalent sample density. The theoretical advantages of hexagonal sampling are well known. However, to the best of our knowledge, the practical benefit of hexagonal sampling in light of modern processing techniques such as RCAN SR is heretofore untested. Our SR system demonstrates a notable advantage of hexagonally sampled imagery when employing a modified RCAN for hexagonal SR.

show abstract

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Zanotta,

Marques,

Motta

et al. 2023

Computers & Geosciences

View full text Add to dashboard Cite

FIFNET: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames

Cited by 10 publications

References 21 publications

Deep learning for anisoplanatic optical turbulence mitigation in long-range imaging

Deep learning for anisoplanatic optical turbulence mitigation in long-range imaging

Resampling and super-resolution of hexagonally sampled images using deep learning

An assisted multi-frame approach for super-resolution in hyperspectral images of rock samples

Contact Info

Product

Resources

About