Marine Vision-Based Situational Awareness Using Discriminative Deep Learning: A Survey

Abstract

The primary task of marine surveillance is to construct a perfect marine situational awareness (MSA) system that serves to safeguard national maritime rights and interests and to maintain blue homeland security. Progress in maritime wireless communication, developments in artificial intelligence, and automation of marine turbines together imply that intelligent shipping is inevitable in future global shipping. Computer vision-based situational awareness provides visual semantic information to human beings that approximates eyesight, which makes it likely to be widely used in the field of intelligent marine transportation. We describe how we combined the visual perception tasks required for marine surveillance with those required for intelligent ship navigation to form a marine computer vision-based situational awareness complex and investigated the key technologies they have in common. Deep learning was a prerequisite activity. We summarize the progress made in four aspects of current research: full scene parsing of an image, target vessel re-identification, target vessel tracking, and multimodal data fusion with data from visual sensors. The paper gives a summary of research to date to provide background for this work and presents brief analyses of existing problems, outlines some state-of-the-art approaches, reviews available mainstream datasets, and indicates the likely direction of future research and development. As far as we know, this paper is the first review of research into the use of deep learning in situational awareness of the ocean surface. It provides a firm foundation for further investigation by researchers in related fields.

Application of Convolutional Neural Network (CNN) to Recognize Ship Structures

“…With the recent development of information and communication technology and artificial intelligence technology, various means of transportation such as airplanes, automobiles, and ships are applying fully automated transportation systems. Nevertheless, a study on the visual interpretation of ships is necessary because the maritime industry is falling behind in this application, unlike other land sectors [ 9 ]. AIS radars and SAR are usually used for obtaining information on ships, but they cannot secure accurate visual cues in real-time, such as the size and structure of ships [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ].…”

“…To facilitate maritime traffic control, the system automatically transmits and receives static information (ship’s name, MMSI, type, and size), dynamic information (ship’s position, speed, heading, and course), and navigation information (ship’s outport, next port, and expected arrival time) between land and ship control offices and between ships and ships. However, since the AIS checks the location of ships in a non-visual way, it does not have the ability to determine their appearance and structure on the deck, which may be a risk factor for drones’ flying and landing [ 7 , 8 , 9 ]. This is an uncertainty factor in the maritime delivery service using drones, which does not guarantee safety.…”

A Hybrid Method for Inland Ship Recognition Using Marine Radar and Closed-Circuit Television

“…Cameras as popular sensors for object recognition are cheap but can offer rich details of targets. The typical cameras are CCTVs for maritime applications [8], whose image-based recognition focuses on feature extraction and where images are converted into multidimensional vectors rather than images that are able to save data storage. Scale-invariant feature transform (SIFT) proposed in 2004, perhaps being a popular method among the mass of descriptors, initially involved extraction stability.…”

Survey on Deep Learning-Based Marine Object Detection

“…Chen et al [10] analyzed the characteristics of the imbalance problem in different kinds of deep detectors and experimentally compared the performance of some state-of-the-art solutions on the COCO benchmark. Qiao et al [11] combined the visual perception tasks required for maritime surveillance with those required for intelligent ship navigation to form a marine computer vision-based situational awareness complex and investigated the key technologies they have in common.…”

“…In recent years, deep learning-based visual perception has been widely applied to autonomous ship navigation and maritime transportation surveillance for intelligent transportation systems (ITS). e survey articles in the maritime field-related applications of computer vision are as follows: Qiao et al [11] summarized the progress made in four aspects: full scene parsing of an image, ship reidentification, ship tracking, and multimodal data fusion with different visual sensors. Prasad et al [16] provided a comprehensive overview of various approaches of video processing for object detection in the maritime environment.…”