A Guide to Image and Video based Small Object Detection using Deep Learning : Case Study of Maritime Surveillance

Abstract

Small object detection (SOD) in optical images and videos is a challenging problem that even state-of-the-art generic object detection methods fail to accurately localize and identify such objects. Typically, small objects appear in real-world due to large camera-object distance. Because small objects occupy only a small area in the input image (e.g., less than 10%), the information extracted from such a small area is not always rich enough to support decision making. Multidisciplinary strategies are being developed by researchers working at the interface of deep learning and computer vision to enhance the performance of SOD deep learning based methods. In this paper, we provide a comprehensive review of over 160 research papers published between 2017 and 2022 in order to survey this growing subject. This paper summarizes the existing literature and provide a taxonomy that illustrates the broad picture of current research. We investigate how to improve the performance of small object detection in maritime environments, where increasing performance is critical. By establishing a connection between generic and maritime SOD research, future directions have been identified. In addition, the popular datasets that have been used for SOD for generic and maritime applications are discussed, and also well-known evaluation metrics for the state-of-the-art methods on some of the datasets are provided.

POSEIDON: A Data Augmentation Tool for Small Object Detection Datasets in Maritime Environments

“…One of these areas pertains to the real-time detection of small vessels, individuals, and other objects in maritime environments using aerial images obtained from drones or small aircraft. Developing robust and precise models for this application would prove to be highly beneficial in search and rescue missions, humanitarian aid efforts, and surveillance [ 2 , 3 , 4 ] and security operations. However, as we have previously noted in our publication [ 5 ], the primary issue is the high cost of capturing such images and the fact that instances in these images tend to be very small.…”

Extended Expectation Maximization for Under-Fitted Models

“…In our previous work, we surveyed numerous strategies employed in deep learning to enhance the performance of small object detection in optical images and videos up to the year 2022 [11]. We showed that beyond the adaptation of newer deep learning structures such as transformers, prevalent approaches include data augmentation, super-resolution, multi-scale feature learning, context learning, attention-based learning, region proposal, loss function regularization, leveraging auxiliary tasks, and spatiotemporal feature aggregation.…”

“…Additionally, we observed that transformers are among the leading methods in localizing small objects across most datasets. However, given that [11] predominantly evaluated over 160 papers focusing on CNN-based networks, an in-depth exploration of transformer-centric methods was not undertaken. Recognizing the growth and exploration pace in the field, there is a timely window now to delve into the current transformer models geared towards small object detection.…”

“…Since 2017, the field has seen the publication of numerous review articles. An extensive discussion and listing of these reviews are presented in our previous survey [11]. Another recent survey article [12] mostly focuses on the CNN-based techniques, too.…”

“…In this survey, we assume the reader is already familiar with generic object detection techniques, their architectures, and relevant performance measures. If the reader requires foundational insight into these areas, we refer the reader to our previous work [11].…”