Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception

Abstract

In this paper, we introduce and tackle the simultaneous enhancement and super-resolution (SESR) problem for underwater robot vision and provide an efficient solution for near real-time applications. We present Deep SESR, a residual-in-residual network-based generative model that can learn to restore perceptual image qualities at 2×, 3×, or 4× higher spatial resolution. We supervise its training by formulating a multi-modal objective function that addresses the chrominance-specific underwater color degradation, lack of image sharpness, and loss in high-level feature representation. It is also supervised to learn salient foreground regions in the image, which in turn guides the network to learn global contrast enhancement. We design an end-to-end training pipeline to jointly learn the saliency prediction and SESR on a shared hierarchical feature space for fast inference. Moreover, we present UFO-120, the first dataset to facilitate large-scale SESR learning; it contains over 1500 training samples and a benchmark test set of 120 samples. By thorough experimental evaluation on the UFO-120 and other standard datasets, we demonstrate that Deep SESR outperforms the existing solutions for underwater image enhancement and super-resolution. We also validate its generalization performance on several test cases that include underwater images with diverse spectral and spatial degradation levels, and also terrestrial images with unseen natural objects. Lastly, we analyze its computational feasibility for single-board deployments and demonstrate its operational benefits for visuallyguided underwater robots. The model and dataset information will be available at: https://github.com/ xahidbuffon/Deep-SESR.

Semantic Segmentation of Underwater Imagery: Dataset and Benchmark

“…These images are carefully chosen from a large pool of samples collected during oceanic explorations and human-robot cooperative experiments in several locations of various water types. We also utilize a few images from large-scale datasets named EUVP [10], USR-248 [27], and UFO-120 [15], which we previously proposed for underwater image enhancement and super-resolution problems. The images are chosen to accommodate a diverse set of natural underwater scenes and various setups for human-robot collaborative experiments.…”

“…Nevertheless, a few recent work [16], [15] explored the performance of contemporary deep CNN-based semantic segmentation models such as VGG-based encoderdecoders [30], [1], UNet [5], and SegNet [3] for underwater imagery. Although they report inspiring results, they only consider sea-grass, sand, and rock as object categories.…”

“…Another genre of approaches for visual attention modeling focuses on spatial saliency computation, i.e., predicting relevance/importance of each pixel in the image [15], [44]. The salient image regions can be exploited for the detection and tracking of specific (known) objects, or for finding new objects of interest in exploration tasks [21].…”

“…In addition to providing more precise object localization, SUIM-Net incurs considerably fewer cases of missed detection, particularly in occluded or low-contrast regions in the image. Moreover, the additional semantic information facilitates much-improved saliency prediction, whereas the class-agnostic models such as Deep SESR [15] and AS-Net [44] concentrate on the high-contrast foreground regions only. These, among many others, are important use cases of the proposed SUIM dataset.…”

“…The existing large-scale annotated data and relevant methodologies are tied to specific applications such as coral-reef classification and coverage estimation [11], [12], fish detection and segmentation [13], [14], etc. Other datasets contain either binary annotations for salient foreground pixels [15] or semantic labels for very few object categories (e.g., sea-grass, rocks/sand, etc.) [16].…”