l, r-Stitch Unit: Encoder-Decoder-CNN Based Image-Mosaicing Mechanism for Stitching Non-Homogeneous Image Sequences

Abstract

Image-stitching (or) mosaicing is considered an active research-topic with numerous use-cases in computer-vision, AR/VR, computer-graphics domains, but maintaining homogeneity among the input image sequences during the stitching/mosaicing process is considered as a primary-limitation & majordisadvantage. To tackle these limitations, this article has introduced a robust and reliable image stitching methodology (l,r-Stitch Unit), which considers multiple non-homogeneous image sequences as input to generate a reliable panoramically stitched wide view as the final output. The l,r-Stitch Unit further consists of a pre-processing, post-processing sub-modules & a l,r-PanoED-network, where each submodule is a robust ensemble of several deep-learning, computer-vision & image-handling techniques. This article has also introduced a novel convolutional-encoder-decoder deep-neural-network (l,r-PanoEDnetwork) with a unique split-encoding-network methodology, to stitch non-coherent input left, right stereo image pairs. The encoder-network of the proposed l,r-PanoED extracts semantically rich deep-featuremaps from the input to stitch/map them into a wide-panoramic domain, the feature-extraction & featuremapping operations are performed simultaneously in the l,r-PanoED's encoder-network based on the split-encoding-network methodology. The decoder-network of l,r-PanoED adaptively reconstructs the output panoramic-view from the encoder networks' bottle-neck feature-maps. The proposed l,r-Stitch Unit has been rigorously benchmarked with alternative image-stitching methodologies on our custom-built traffic dataset and several other public-datasets. Multiple evaluation metrics (SSIM, PSNR, MSE, L α,β,γ , FM-rate, Average-latency-time) & wild-Conditions (rotational/color/intensity variances, noise, etc) were considered during the benchmarking analysis, and based on the results, our proposed method has outperformed among other image-stitching methodologies and has proved to be effective even in wild non-homogeneous inputs.INDEX TERMS Deep feature extraction, encoder-decoder cnn, image mosaicing, multi-image registration, non-homogeneous image stitching.

Content-Seam-Preserving Multi-Alignment Network for Visual-Sensor-Based Image Stitching

“…Inspired by the idea of deep homography technology, the deep image-stitching method has been proposed to deal with visual-sensor-based images [37,38]. Nie et al [39] proposed an image-stitching network via global homography to eliminate image artifacts.…”

Semi-Supervised Image Stitching from Unstructured Camera Arrays

“…Song et al used CNNs in [16,18], making use of weak supervision and expanding their network to work with images taken in a simulated outdoor environment, which can be more difficult as these images have more variation in exposure levels. In [15], Chilukuri et al stitched two images together and leveraged auto-encoders [21] in addition to standard convolutional layers when constructing their network. Specifically, they encoded two input images into a shared space and then decoded the result into a single output image.…”

“…This is because SandFall fixes the input size regardless of the number of cameras. As a result, our method is much more scalable for systems with many cameras compared to other methods, which often need on the order of one second to stitch an image pair together even when using GPU acceleration [15,38]. It is important to note that we ran our method entirely on CPU in our experiments in order to more directly compare our approach to multi-band blending and seam-based stitching, which are recent approaches able to operate on unstructured camera arrays and are limited to CPU.…”

“…A few recent works have examined image stitching using deep learning [15][16][17][18]. However, all of these focused on pairwise stitching, wherein two inputs are stitched together and the output is stitched with the next input until all inputs have been used, similar to the traditional methods discussed above.…”

A Novel Fast Image Stitching Method Based on the Combination of SURF and Cell

“…is article has introduced a robust and reliable image stitching methodology (l, r-Stitch Unit), which considers multiple nonhomogeneous image sequences as input to generate a reliable panoramically stitched wide view as the final output. Besides, it has also introduced a novel convolutional-encoder-decoder deepneural-network (l, r-PanoED-network) with a unique splitencoding-network methodology, to stitch noncoherent input left, right stereo image pairs [17]. Xue et al proposed a high-quality fisheye image stitching algorithm (LLBI-AW).…”