Instance Segmentation for Large, Multi-Channel Remote Sensing Imagery Using Mask-RCNN and a Mosaicking Approach

Abstract

Instance segmentation is the state-of-the-art in object detection, and there are numerous applications in remote sensing data where these algorithms can produce significant results. Nevertheless, one of the main problems is that most algorithms use Red, Green, and Blue (RGB) images, whereas Satellite images often present more channels that can be crucial to improve performance. Therefore, the present work brings three contributions: (a) conversion system from ground truth polygon data into the Creating Common Object in Context (COCO) annotation format; (b) Detectron2 software source code adaptation and application on multi-channel imagery; and (c) large scene image mosaicking. We applied the procedure in a Center Pivot Irrigation System (CPIS) dataset with ground truth produced by the Brazilian National Water Agency (ANA) and Landsat-8 Operational Land Imager (OLI) imagery (7 channels with 30-m resolution). Center pivots are a modern irrigation system technique with massive growth potential in Brazil and other world areas. The round shapes with different textures, colors, and spectral behaviors make it appropriate to use Deep Learning instance segmentation. We trained the model using 512 × 512-pixel sized patches using seven different backbone structures (ResNet50- Feature Pyramid Network (FPN), Resnet50-DC5, ResNet50-C4, Resnet101-FPN, Resnet101-DC5, ResNet101-FPN, and ResNeXt101-FPN). The model evaluation used standard COCO metrics (Average Precision (AP), AP50, AP75, APsmall, APmedium, and AR100). ResNeXt101-FPN had the best results, with a 3% advantage over the second-best model (ResNet101-FPN). We also compared the ResNeXt101-FPN model in the seven-channel and RGB imagery, where the multi-channel model had a 3% advantage, demonstrating great improvement using a larger number of channels. This research is also the first with a mosaicking algorithm using instance segmentation models, where we tested in a 1536 × 1536-pixel image using a non-max suppression sorted by area method. The proposed methodology is innovative and suitable for many other remote sensing problems and medical imagery that often present more channels.

Dealing With Clouds and Seasonal Changes for Center Pivot Irrigation Systems Detection Using Instance Segmentation in Sentinel-2 Time Series

“…1: Representation of the seasonal variations among center pivots in a Sentinel-2 image using the Red, Green, and Blue spectral bands. approaches: (a) detection of the core point of the center pivot [47]; (b) object detection with the establishment of bounding boxes around CPIS [20], [57]; (c) semantic segmentation that performs a pixel-wise classification where all CPIS pixels receive a label [18], [58], [59]; and (d) instance segmentation that produces bounding boxes and pixel-wise segmentation masks on CPIS [17].…”

“…However, Detectron2's Mask-RCNN algorithm requires labels in the COCO annotation format [67], in which each image tile needs a JSON file with the corresponding annotations. Thus, we applied the method used by de Carvalho et al [17] to convert polygonal GIS data into the instance segmentation annotation format. Each object acquired a unique value from 1 to N, with N being the total number of CPIS.…”

“…Among the instance segmentation models, the Mask-RCNN is the most common approach.The Mask-RCNN algorithm has three objectives: (a) identify the bounding box for a given object, (b) classify that bounding box according to the object's [68] is one of the most efficient instance segmentation frameworks, introduced by the Facebook Artificial Intelligence Research (FAIR), powered by Pytorch. This architecture, usually applied to traditional RGB imagery, requires adjustments to be compatible with the remote sensing data [17]. Hence, we use the Detectron2 software from the Pytorch library with some adaptations to suit our purposes.…”

“…In this context, the National Water Agency (ANA -Agência Nacional deÁguas) performs the annual CPIS mapping based on remote sensing and visual interpretation images [15], [16]. In the quest to automate CPIS detection, methods based on Deep Learning (DL) have achieved results with metrics over 90% [17], [18], [19], [20]. This approach allows for several advantages such as lower costs, faster, and more accurate surveying when compared to visual interpretation and traditional machine learning methods.…”

Remote Sensing for Monitoring Photovoltaic Solar Plants in Brazil Using Deep Semantic Segmentation

“…This approach is an effective alternative to construction inspection, requiring periodic data and free satellite imagery. Previous studies on PV panel detection have not yet shown reasonable solutions for classifying large regions, and the use of mosaicking with sliding windows is a promising solution [85][86][87].…”