Image and video upscaling from local self-examples

Abstract

We propose a new high-quality and efficient single-image upscaling technique that extends existing example-based super-resolution frameworks. In our approach we do not rely on an external example database or use the whole input image as a source for example patches. Instead, we follow a local self-similarity assumption on natural images and extract patches from extremely localized regions in the input image. This allows us to reduce considerably the nearest-patch search time without compromising quality in most images. Tests, that we perform and report, show that the local-self similarity assumption holds better for small scaling factors where there are more example patches of greater relevance. We implement these small scalings using dedicated novel non-dyadic filter banks, that we derive based on principles that model the upscaling process. Moreover, the new filters are nearly-biorthogonal and hence produce high-resolution images that are highly consistent with the input image without solving implicit back-projection equations. The local and explicit nature of our algorithm makes it simple, efficient and allows a trivial parallel implementation on a GPU. We demonstrate the new method ability to produce high-quality resolution enhancement, its application to video sequences with no algorithmic modification, and its efficiency to perform real-time enhancement of lowresolution video standard into recent high-definition formats.

Single-Image Super-Resolution: A Benchmark

“…Broadly speaking, SISR algorithms can be categorized based on their tasks. While domain-specific SISR algorithms focus on specific classes of images such as faces [35,42], scenes [33], and graphics artwork [18], generic SISR algorithms [10,38,8,3,27,30,11,46,9,34,32,12,5,39,44,43] are developed for all kinds of images where the priors are typically based on primitive image properties such as edges and segments. In order to evaluate the performance of a SISR algorithm, human subject studies or ground truth images are used [33,43].…”

Acceleration techniques and evaluation on multi-core CPU, GPU and FPGA for image processing and super-resolution

“…In the field of SuperResolution, Freedman et al [8] applied their single-image SR method on a GTX 480 GPU to achieve near realtime performance when upsampling from 640 × 360 to 1920 × 1080. Using dictionary-based methods, the authors of [9] also provide a GPU implementation of their deformable image patch method.…”

“…For image size 1120×1120, D = 200 disparity levels, and stripe size 1120×28, we get a HW cost of 998 slices (3%), 2978 LUTs (2%), 3116 registers (1%), 0 DSPs, and 101 RAMB36 (24%). If we enlarge the aggregation mask from 7 × 7 to 13 × 13, the HW cost will increase to 2690 slices (8,492 LUTs and 8,640 registers) showing that the most expensive component is the parallel arithmetic circuit performing aggregation (depending on mask size, aggregation consumes 60%-85% of the utilized logic resources). Achieving up to 344 MHz clock frequency, a single Disparity module will process the entire 1120×1120 image pair in 1.87 sec (time is almost independent of the mask size due to the applied full mask parallelization).…”

Multi-example feature-constrained back-projection method for image super-resolution

“…NeedFS [25] is based on neighbor embedding edge detection feature selection. IUIE [26] and NARM [27] represent example-based and sparse coding methods respectively. FCME is our multi-example feature-constrained method ignoring back-projection.…”