Optimal Seamline Detection for Orthoimage Mosaicking by Combining Deep Convolutional Neural Network and Graph Cuts

Abstract

When mosaicking orthoimages, especially in urban areas with various obvious ground objects like buildings, roads, cars or trees, the detection of optimal seamlines is one of the key technologies for creating seamless and pleasant image mosaics. In this paper, we propose a new approach to detect optimal seamlines for orthoimage mosaicking with the use of deep convolutional neural network (CNN) and graph cuts. Deep CNNs have been widely used in many fields of computer vision and photogrammetry in recent years, and graph cuts is one of the most widely used energy optimization frameworks. We first propose a deep CNN for land cover semantic segmentation in overlap regions between two adjacent images. Then, the energy cost of each pixel in the overlap regions is defined based on the classification probabilities of belonging to each of the specified classes. To find the optimal seamlines globally, we fuse the CNN-classified energy costs of all pixels into the graph cuts energy minimization framework. The main advantage of our proposed method is that the pixel similarity energy costs between two images are defined using the classification results of the CNN based semantic segmentation instead of using the image informations of color, gradient or texture as traditional methods do. Another advantage of our proposed method is that the semantic informations are fully used to guide the process of optimal seamline detection, which is more reasonable than only using the hand designed features defined to represent the image differences. Finally, the experimental results on several groups of challenging orthoimages show that the proposed method is capable of finding high-quality seamlines among urban and non-urban orthoimages, and outperforms the state-of-the-art algorithms and the commercial software based on the visual comparison, statistical evaluation and quantitative evaluation based on the structural similarity (SSIM) index.

Automatic Seam-Line Detection in UAV Remote Sensing Image Mosaicking by Use of Graph Cuts

“…However, due to the influence of image matching errors, image interpolation errors and intrinsic factors such as illumination and noise, it is difficult to avoid misplacement, ghosting, and aliasing. The other is the mosaic methods based on seam-line, their solution is different from the mosaic methods based on pixel fusion [18][19][20][21][22]. The mosaic methods based on seam-line only take the pixels of the reference image or pre-mosaicked image in the mosaicked result, that is, the overlapped area is divided into two blocks, one for taking pixels from the reference image and the other for taking pixels from the pre-mosaicked image.…”

“…Because using dynamic programming to obtain a local optimal seam-line cannot reach a good result. Many scholars have studied how to use a search method to obtain the global optimal seam-line [20][21][22]. Among them, the optimal seam-line search method based on graph cuts is the most representative.…”

“…It uses graph cuts to obtain the best mosaic texture on both sides of the seam-line to obtain the optimal seam-line, which can solve the shortcomings of dynamic programming in the image mosaicking (such as no need to specify additional energy end points, etc.). In addition, compared with the annealing algorithm and ant colony algorithm, it has the advantage of fast calculation [20][21][22][23][24][25][26]. In these optimal seam-line search methods based on graph cuts, they can be divided into a color domain-based method, a gradient-based method, and a color domain and gradient domain-based method according to the difference of the energy function smoothing term.…”

Automatic Seamline Determination for Urban Image Mosaicking Based on Road Probability Map from the D-LinkNet Neural Network

“…Based on the integrated deep convolutional neural network (CNN) and graph cuts energy minimization framework, Li [26] proposed a novel algorithm to optimize seamlines for image mosaicking. Different from the previous method [22], this method defined similarity energy terms of the graph cut using the semantic classification classified by the CNN instead of using the color, gradient, or texture.…”

Learnable Gated Convolutional Neural Network for Semantic Segmentation in Remote-Sensing Images

“…Technically, it is natural to apply those existing approaches for natural images to RS images. The efforts have been actually made by researchers with comparative studies [30][31][32][33]. Such practices have demonstrated the fact that directly using the existing approaches for natural images cannot guarantee to yield satisfactory segmentations for RS images.…”