Cross-Modal Progressive Comprehension for Referring Segmentation

Abstract

Given a natural language expression and an image/video, the goal of referring segmentation is to produce the pixellevel masks of the entities described by the subject of the expression. Previous approaches tackle this problem by implicit feature interaction and fusion between visual and linguistic modalities in a one-stage manner. However, human tends to solve the referring problem in a progressive manner based on informative words in the expression, i.e., first roughly locating candidate entities and then distinguishing the target one. In this paper, we propose a Cross-Modal Progressive Comprehension (CMPC) scheme to effectively mimic human behaviors and implement it as a CMPC-I (Image) module and a CMPC-V (Video) module to improve referring image and video segmentation models. For image data, our CMPC-I module first employs entity and attribute words to perceive all the related entities that might be considered by the expression. Then, the relational words are adopted to highlight the target entity as well as suppress other irrelevant ones by spatial graph reasoning. For video data, our CMPC-V module further exploits action words based on CMPC-I to highlight the correct entity matched with the action cues by temporal graph reasoning. In addition to the CMPC, we also introduce a simple yet effective Text-Guided Feature Exchange (TGFE) module to integrate the reasoned multimodal features corresponding to different levels in the visual backbone under the guidance of textual information. In this way, multi-level features can communicate with each other and be mutually refined based on the textual context. Combining CMPC-I or CMPC-V with TGFE can form our image or video version referring segmentation frameworks and our frameworks achieve new state-of-the-art performances on four referring image segmentation benchmarks and three referring video segmentation benchmarks respectively. Our code is available at https://github.com/spyflying/CMPC-Refseg.

Phrase-Based Affordance Detection via Cyclic Bilateral Interaction

“…Yu et al [34] proposed a modular network which decomposes the input natural language description into subject, location, and relationship attributes to improve the localization performance. Liu et al [35] adopted graph models with an attention mechanism to capture the relationship between the object regions in the given image. In association with visual affordance, Mi et al [36], [37] investigated the use of natural language to guide visual affordance detection.…”

“…Hu et al [22] designed a bi-directional relationship inferring network to model the relationship between linguistic and visual features. Liu et al [35] proposed a model that first perceives all the entities in the image according to the entity and attribution words in the expression, then infers the location of the target object with the words that represent the relationship. Jing et al [23] first gets the position prior of the referring object based on the language and image, then generates segmentation mask based on the position prior.…”

“…Secondly, our phrases only describe affordance without presenting entity words. Therefore, it is unable to utilize relationships between entities as in [23], [35], [40] to localize objects. We adopt short phrases to describe affordances rather than long sentences to meet practical scenarios.…”

“…We adopt short phrases to describe affordances rather than long sentences to meet practical scenarios. This disables us to leverage the text context information to capture the relationship between linguistic and visual features like the operations in [20], [22], [35], [40]. The work mentioned above also utilized natural language to learn visual affordances [36], [37].…”

“…where i is the elements of the ground-truth mask and H × W denotes the size of the ground-truth mask. [49] , CPD [50], OSAD [14], OAFFD [13], PSPNET [51], DEEPLABV3+ (DLABV3+) [52] CMSA [40], BRINET [22], CMPC [35]) IN TERMS OF FIVE METRICS (IOU (↑), F β (↑), E φ (↑), CC (↑), AND MAE (↓)). BOLD AND UNDERLINE INDICATE THE BEST AND THE SECOND-BEST SCORES, RESPECTIVELY.…”