Referring Image Segmentation via Cross-Modal Progressive Comprehension

Huang, Shaofei; Hui, Tianrui; Liu, Si; Li, Guanbin; Wei, Yunchao; Han, Jizhong; Liu, Luoqi; Li, Bo

doi:10.1109/cvpr42600.2020.01050

Cited by 134 publications

(90 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…referred object), which is specified by the given natural language description. Compared with CMPC [13], a state-of-the-art method for RIS, which fails to generate precise or even correct prediction masks for unsalient referents as shown in (b), our proposed TV-Net can produce segmentation masks as shown in (c), which are highly consistent with the ground-truth in (d).…”

Section: Introductionmentioning

confidence: 88%

“…As graph neural network [35,42] presents a new form of mining the relationship between data, Hui et al [15] and Yang et al introduce graph structure models to achieve efficient message passing in RIS. Moreover, some works [13,15] also consider the linguistic roles of each word during multimodal interaction process. Words are classified into four categories, and a progressive comprehension process is proposed under the guidance of different type of words in [13].…”

Section: Related Workmentioning

confidence: 99%

“…As for language feature, we encode each word in sentence T with a LSTM to generate language feature s ∈ R 𝑑 𝑠 , where 𝑑 𝑠 is the dimension of language embedding space. Following prior works [12,13,37], an 8-D spatial coordinate feature 𝑂 ∈ R ℎ 𝑙 ×𝑤 𝑙 ×8 is integrated into the enriched visual features 𝑉 ′ 𝑙 before cross-modal fusion. As it is commonly assumed that the language refers to highlevel visual concepts while leaving low-level visual processing unaffected [34], it is natural to aggregate language feature with high-level visual features (i.e., the low-resolution feature maps).…”

Section: Cross-modal Fusionmentioning

confidence: 99%

“…Referring Image Segmentation (RIS) [3,12,13,37] has emerged as a prominent multi-discipline research problem. Unlike traditional semantic segmentation that segments the objects of predefined classes, RIS aims to accurately segment the object specified by one natural language expression.…”

Section: Introductionmentioning

confidence: 99%

“…Despite remarkable progresses have been made with the existing cross-modal learning mechanisms, the segmentation results still remain unsatisfactory, especially when the visual cues of referents (i.e.,referred objects) are insufficient. As shown in Figure 1, given the natural language expression "small table next to the chair", CMPC [13], the state-of-the-art method, is not able to produce a precise segmentation mask, while for "table left bottom", CMPC Figure 1: Referring image segmentation (RIS) aims to segment the referent (i.e. referred object), which is specified by the given natural language description.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Two-stage Visual Cues Enhancement Network for Referring Image Segmentation

Jiao

Jie

Luo

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Referring Image Segmentation (RIS) aims at segmenting the target object from an image referred by one given natural language expression. The diverse and flexible expressions as well as complex visual contents in the images raise the RIS model with higher demands for investigating fine-grained matching behaviors between words in expressions and objects presented in images. However, such matching behaviors are hard to be learned and captured when the visual cues of referents (i.e. referred objects) are insufficient, as the referents with weak visual cues tend to be easily confused by cluttered background at boundary or even overwhelmed by salient objects in the image. And the insufficient visual cues issue can not be handled by the cross-modal fusion mechanisms as done in previous work. In this paper, we tackle this problem from a novel perspective of enhancing the visual information for the referents by devising a Two-stage Visual cues enhancement Network (TV-Net), where a novel Retrieval and Enrichment Scheme (RES) and an Adaptive Multi-resolution feature Fusion (AMF) module are proposed. Specifically, RES retrieves the most relevant image from an external data pool with regard to both the visual and textual similarities, and then enriches the visual information of the referent with the retrieved image for better multimodal feature learning. AMF further enhances the visual detailed information by incorporating the highresolution feature maps from lower convolution layers of the image. Through the two-stage enhancement, our proposed TV-Net enjoys better performances in learning fine-grained matching behaviors between the natural language expression and image, especially when the visual information of the referent is inadequate, thus produces better segmentation results. Extensive experiments are conducted to validate the effectiveness of the proposed method on the RIS task, with our proposed TV-Net surpassing the state-of-theart approaches on four benchmark datasets. Our code is available at: https://github.com/SxJyJay/TV-Net.

show abstract

Section: Introductionmentioning

confidence: 88%