Very Long Natural Scenery Image Prediction by Outpainting

Abstract

Comparing to image inpainting, image outpainting receives less attention due to two challenges in it. The first challenge is how to keep the spatial and content consistency between generated images and original input. The second challenge is how to maintain high quality in generated results, especially for multi-step generations in which generated regions are spatially far away from the initial input. To solve the two problems, we devise some innovative modules, named Skip Horizontal Connection and Recurrent Content Transfer, and integrate them into our designed encoder-decoder structure. By this design, our network can generate highly realistic outpainting prediction effectively and efficiently. Other than that, our method can generate new images with very long sizes while keeping the same style and semantic content as the given input. To test the effectiveness of the proposed architecture, we collect a new scenery dataset with diverse, complicated natural scenes. The experimental results on this dataset have demonstrated the efficacy of our proposed network. The code and dataset are available from https: //github.com/z-x-yang/NS-Outpainting.

Scenery Image Extension via Inpainting With a Mirrored Input

“…Our contributions are summarized as follows: 1) We propose a mirrored input, which replaces generative image extension with an image inpainting problem and thus helps to achieve higher quality pixel gener- [1], [2] predict semantic content in the surroundings of the target image. Some methods [9], [10] perform extrapolation on texture patterns.…”

“…Dataset. We use the Places subset [29] and Scenery dataset proposed in VLNS [2]. We construct the Places subset from the Places365-Challenge [29] dataset's trainset.…”

“…We used the following 20 classes to create the Places subset: aquarium, athletic_field/outdoor, beach, cliff, coast, forest_path, golf_course, harbor, lake/natural, mountain, ocean, pier, pond, rainforest, river, skyscraper, swamp, underwater/ocean_deep, valley, vegetable_garden. The Scenery dataset consists of 5000 training and 1040 test images, which is different from the actual dataset combination used in VLNS [2]. While most of the Scenery datasets are images of mountains, the Places subset is composed of images with diverse categories, and thus it is relatively difficult [30]- [32] to stably train a generator with the Places subset.…”

“…Many recent image inpainting methods propose new convolution methods to address free-form and small missing regions. However, because the missing regions in this work are rectangular and fixed large holes, the latest methods are not suitable, as claimed in VLNS [2]. As shown by GLCIC, natural image completion requires global and local consistency, so it is necessary to deal with global features, such as the scene context and semantics, as well as local features, such as texture and structure.…”

“…One of the reasons for this is that the position of the pixel to be generated is far from the input image pixels, and therefore the information is not sufficiently propagated by convolution. To address this issue, VLNS [2] has proposed a module to effectively propagate information in the horizontal direction. However, because the content is generated as if the same contents are repeated, the scenes that can be handled are limited (e.g., images of mountain ranges).…”