Super-resolved free-viewpoint image synthesis combined with sparse-representation-based super-resolution

Abstract

We consider super-resolved free-viewpoint image synthesis (SR-FVS), where a high-resolution (HR) image that would be observed from a virtual viewpoint is synthesized from a set of low-resolution multi-view images. In previous studies, methods for SR-FVS were proposed on the basis of reconstruction-based super-resolution (RB-SR). RB-SR uses multiple images to synthesize an HR image and thereby can naturally be applied to SR-FVS, where multi-view images are given as the input. However, the quality of the synthesized image depends on observation conditions such as the depth of the target scene, so sometimes the quality of SR-FVS can degrade severely. To mitigate such degradation, we propose integrating learning-based super-resolution (LB-SR), which uses knowledge learned from massive natural images, into the SR-FVS process. In this paper, we adopt sparse coding super-resolution (ScSR) as a LB-SR method and combine ScSR with an existing SR-FVS method.

A depth-based Multi-view Super-Resolution Method Using Image Fusion and Blind Deblurring

“…The comparisons of MSE, PSNR, and SSIM are shown in Table 5. Compared with the algorithms in [13,14], the proposed algorithm achieves the smallest MSE, the highest PSNR, and the most significant SSIM, thereby indicating the competitiveness of the proposed method.…”

“…In this section, we report the detailed experimental results of the proposed method, and compare the results of the proposed method with those of the SR-FVS [13] and the ScSR + SR-FVS [14] methods. All the experiments were performed using MATLAB on a 64 bit Windows 8 personal computer with an Intel Core i7 3.6 GHz processor and 16 GB RAM.…”

“…A super-resolution-free-viewpoint image synthesis (SR-FVS) method that used adaptive regularization for MVSR to address depth inaccuracies was proposed by Takahashi; this method simultaneously realized free-viewpoint image SR and free-viewpoint depth estimation [13]. Nakashima et al [14] combined a learning-based SR method, namely, sparse coding SR (ScSR) [15], with an existing SR-FVS method [13] to improve the quality of the desired HR image of the target viewpoint.…”

“…SR reconstruction was divided into two stages in [2,13,14]: image fusion (IF), which was called image blend in [13,14], and maximum-a-posteriori (MAP). In the MAP stage, however, [13] ignored the effect of blur, whereas [2,14] assumed that blur was known. Consequently, all the energy functions constructed in the MAP stage had no regularization term for blur.…”

“…In fact, the blur kernel was typically unknown in practical situations. In this study, we have adopted the two-stage SR framework in [2,13,14], and considered the effect of unknown blur (i.e., a regularization term for blur has been added into the energy function in the MAP stage). We have actually reformulated the MVSR problem into an IF problem and a blind deblurring (BD) problem.…”