RGB-D Salient Object Detection with Cross-Modality Modulation and Selection

Abstract

We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD). The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features. First, we propose a cross-modality feature modulation (cmFM) module to enhance feature representations by taking the depth features as prior, which models the complementary relations of RGB-D data. Second, we propose an adaptive feature selection (AFS) module to select saliency-related features and suppress the inferior ones. The AFS module exploits multi-modality spatial feature fusion with the self-modality and cross-modality interdependencies of channel features are considered. Third, we employ a saliency-guided position-edge attention (sg-PEA) module to encourage our network to focus more on saliency-related regions. The above modules as a whole, called cmMS block, facilitates the refinement of saliency features in a coarse-to-fine fashion. Coupled with a bottom-up inference, the refined saliency features enable accurate and edge-preserving SOD. Extensive experiments demonstrate that our network outperforms stateof-the-art saliency detectors on six popular RGB-D SOD benchmarks.

Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection

“…The LFSD dataset [27] includes 100 RGB-D images via a light field camera. Following [26,33], we adopt 2985 images as our training data, including 1485 samples from NJUD, 700 samples from NLPR, and 800 samples from DUT, and all the remaining images are used as testing. Evaluation metrics.…”

BTS-Net: Bi-Directional Transfer-And-Selection Network for RGB-D Salient Object Detection

“…The utilization of RGB-D data for SOD has been extensively explored for years. Traditional methods rely on hand-crafted features [25][26][27][28], while recently, deep learning-based methods have made great progress [5][6][7][8][10][11][12][13][14][15][16][17][18][19][20][21]. Based on the scope of this paper, we divide existing deep-based models into two types according to how they extract RGB and depth features, namely: parallel independent encoders (Fig.…”

“…Zhang et al [12] introduced a complimentary interaction module to select useful features. In [13], Li et al enhanced feature representations by taking depth features as priors. Chen et al [14] proposed to extract depth features with a light-weight depth branch and conduct progressive refinement.…”

“…To demonstrate the effectiveness of the proposed method, we compare it with 16 state-of-the-art (SOTA) methods, i.e. : PCF [10], MMCI [17], CPFP [16], DMRA [11], D3Net [20], SSF [12], A2dele [18], UCNet [7], JLDCF [6], cmMS [13], CoNet [19], PGAR [14], Cas-Gnn [31], DANet [21], HDFNet [8], BBSNet [5]. Quantitative results are shown in Table 1.…”

“…Additional useful spatial information embedded in depth maps could somewhat help overcome the aforementioned challenges. Although a lot of advances [9] have been made in this field by exploring cross-modal complementarity [5][6][7][8][10][11][12][13][14][15][16][17][18][19][20][21][22], we notice that existing models are still insufficient on extracting robust saliency features. As shown in Fig.…”