Two-stage Visual Cues Enhancement Network for Referring Image Segmentation

Abstract

Referring Image Segmentation (RIS) aims at segmenting the target object from an image referred by one given natural language expression. The diverse and flexible expressions as well as complex visual contents in the images raise the RIS model with higher demands for investigating fine-grained matching behaviors between words in expressions and objects presented in images. However, such matching behaviors are hard to be learned and captured when the visual cues of referents (i.e. referred objects) are insufficient, as the referents with weak visual cues tend to be easily confused by cluttered background at boundary or even overwhelmed by salient objects in the image. And the insufficient visual cues issue can not be handled by the cross-modal fusion mechanisms as done in previous work. In this paper, we tackle this problem from a novel perspective of enhancing the visual information for the referents by devising a Two-stage Visual cues enhancement Network (TV-Net), where a novel Retrieval and Enrichment Scheme (RES) and an Adaptive Multi-resolution feature Fusion (AMF) module are proposed. Specifically, RES retrieves the most relevant image from an external data pool with regard to both the visual and textual similarities, and then enriches the visual information of the referent with the retrieved image for better multimodal feature learning. AMF further enhances the visual detailed information by incorporating the highresolution feature maps from lower convolution layers of the image. Through the two-stage enhancement, our proposed TV-Net enjoys better performances in learning fine-grained matching behaviors between the natural language expression and image, especially when the visual information of the referent is inadequate, thus produces better segmentation results. Extensive experiments are conducted to validate the effectiveness of the proposed method on the RIS task, with our proposed TV-Net surpassing the state-of-theart approaches on four benchmark datasets. Our code is available at: https://github.com/SxJyJay/TV-Net.

Improving Panoptic Narrative Grounding by Harnessing Semantic Relationships and Visual Confirmation

“…One-stage frameworks (Suo et al 2021;Li and Sigal 2021;Hu, Rohrbach, and Darrell 2016) have been proposed. To model semantic relationships between vision and language, recent methods (Ding et al 2021;Feng et al 2021;Jiao et al 2021;Li and Sigal 2021;Yang et al 2022;Luo et al 2020b)incorporate complex cross-attention mechanisms inspired by the powerful abilities of Transformers (Vaswani et al 2017) for capturing long-range dependencies.…”

Boosting the Transferability of Video Adversarial Examples via Temporal Translation

“…Recent years have witnessed the great success of deep learning techniques on a series of tasks (He et al 2016;Liu et al 2018;Feng et al 2021), such as image recognition (He et al 2016;Liu et al 2020;Chen et al 2020b,a), Image segmentation (Jiao et al 2021), object detection (Ren et al 2016), video recognition and retrieval (Wu et al 2020c;Song et al 2021). Therefore, DNNs have been widely applied in realworld applications, e.g., online recognition services, navigation robots, autonomous driving (Tian et al 2018), etc.…”