Real-Time Ship Segmentation in Maritime Surveillance Videos Using Automatically Annotated Synthetic Datasets

Abstract

This work proposes a new system capable of real-time ship instance segmentation during maritime surveillance missions by unmanned aerial vehicles using an onboard standard RGB camera. The implementation requires two stages: an instance segmentation network able to produce fast and reliable preliminary segmentation results and a post-processing 3D fully connected Conditional Random Field, which significantly improves segmentation results by exploring temporal correlations between nearby frames in video sequences. Moreover, due to the absence of maritime datasets consisting of properly labeled video sequences, we create a new dataset comprising synthetic video sequences of maritime surveillance scenarios (MarSyn). The main advantages of this approach are the possibility of generating a vast set of images and videos, being able to represent real-world scenarios without the necessity of deploying the real vehicle, and automatic labels, which eliminate human labeling errors. We train the system with the MarSyn dataset and with aerial footage from publicly available annotated maritime datasets to validate the proposed approach. We present some experimental results and compare them to other approaches, and we also illustrate the temporal stability provided by the second stage in missing frames and wrong segmentation scenarios.

POSEIDON: A Data Augmentation Tool for Small Object Detection Datasets in Maritime Environments

“…One of these areas pertains to the real-time detection of small vessels, individuals, and other objects in maritime environments using aerial images obtained from drones or small aircraft. Developing robust and precise models for this application would prove to be highly beneficial in search and rescue missions, humanitarian aid efforts, and surveillance [ 2 , 3 , 4 ] and security operations. However, as we have previously noted in our publication [ 5 ], the primary issue is the high cost of capturing such images and the fact that instances in these images tend to be very small.…”

Use of Synthetic Data in Maritime Applications for the Problem of Steam Turbine Exergy Analysis

“…Synthetic data has also been applied on ship wake detection, for data fusion from multiple sensors by Higgins et al (2022) [11], showing an improvement in results when real data is bolstered by synthetically generated ones. While a wider application of synthetic data in maritime applications focuses on images [12,13], there is a clear lack of application of synthetic data generators which create additional data points based on statistical methods. The current state of research into applying ML for exergy/energy analysis shows that the field is active.…”

LI-DWT- and PD-FC-MSPCNN-Based Small-Target Localization Method for Floating Garbage on Water Surfaces

“…For example, Arshad [11] proposed an algorithm capable of effectively detecting and monitoring multiple ships in real time using morphological operations and edge information to segment and locate ships for ship detection and tracking, in addition to a smoothing filter and Sobel operator for edge detection. Imtiaz et al [12] used the intensity and temporal probability maps of input image frames; these are then combined to determine the threshold for segmenting driftwood targets in water using the temporal connection method, thereby effectively overcoming the effects of illumination changes and wave interference for rapid driftwood target detection in videos. Similarly, Ribeiro et al [13] used the YOLACT++ segmentation network with ResNet50 as the backbone feature extraction network and implemented a 3D constant rare factor (CRF) to improve frame loss and ensure the temporal stability of the model.…”