Improving One-Stage Visual Grounding by Recursive Sub-query Construction

Yang, Zhengyuan; Chen, Tianlang; Wang, Liwei; Luo, Jiebo

doi:10.1007/978-3-030-58568-6_23

Cited by 139 publications

(142 citation statements)

References 42 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Some recent approaches adopt one-stage frameworks to tackle referring expression object segmentation [3], zeroshot grounding [30], and visual grounding [41,40,39], where the language features are fused with the object detector. In these methods, the models are end-to-end trainable and more computationally efficient.…”

Section: Related Workmentioning

confidence: 99%

Section: Evaluation On Seen Datasetsmentioning

confidence: 99%

“…Table 1 shows the results of the proposed algorithm against state-of-the-art methods [27,25,44,21,48,45,42,22,3,37,41,19,40]. All the compared approaches except for the recent methods [3,41,19,40] adopt two-stage frameworks, where the prediction is chosen from a set of proposals. Therefore, their models are not end-to-end trainable, while our one-stage framework is able to learn better feature representations by end-to-end training.…”

Section: Evaluation On Seen Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

Understanding Synonymous Referring Expressions via Contrastive Features

Chen¹,

Tsai²,

Yang³

2021

Preprint

View full text Add to dashboard Cite

Referring expression comprehension aims to localize objects identified by natural language descriptions. This is a challenging task as it requires understanding of both visual and language domains. One nature is that each object can be described by synonymous sentences with paraphrases, and such varieties in languages have critical impact on learning a comprehension model. While prior work usually treats each sentence and attends it to an object separately, we focus on learning a referring expression comprehension model that considers the property in synonymous sentences. To this end, we develop an end-to-end trainable framework to learn contrastive features on the image and object instance levels, where features extracted from synonymous sentences to describe the same object should be closer to each other after mapping to the visual domain. We conduct extensive experiments to evaluate the proposed algorithm on several benchmark datasets, and demonstrate that our method performs favorably against the state-of-the-art approaches. Furthermore, since the varieties in expressions become larger across datasets when they describe objects in different ways, we present the cross-dataset and transfer learning settings to validate the ability of our learned transferable features.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Evaluation On Seen Datasetsmentioning

confidence: 99%

Section: Evaluation On Seen Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Understanding Synonymous Referring Expressions via Contrastive Features

Chen¹,

Tsai²,

Yang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While achieving a large-margin improvement compared with that try to directly regress object from the entire image, such progress may attribute to the robust local features representation on the grid. Another line to improve one-stage visual grounding is to apply complex language modeling, such as decomposing the longer phrase into multiple parts [38]. In this work, we do not use complex techniques for language modeling.…”

Section: Related Workmentioning

confidence: 99%

“…Eq.6 lution in v to obtain feature maps with the same channel c v and concatenate the coordinate features with a 8 dimension position embedding vector, which is the same with prior work [39,38], such that we generate the fused feature maps…”

Section: Landmark Feature Convolution Modulementioning

confidence: 99%

Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding

Huang

Lian

Luo

et al. 2021

Preprint

View full text Add to dashboard Cite

An LBYL ('Look Before You Leap') Network is proposed for end-to-end trainable one-stage visual grounding. The idea behind LBYL-Net is intuitive and straightforward: we follow a language's description to localize the target object based on its relative spatial relation to 'Landmarks', which is characterized by some spatial positional words and some descriptive words about the object. The core of our LBYL-Net is a landmark feature convolution module that transmits the visual features with the guidance of linguistic description along with different directions. Consequently, such a module encodes the relative spatial positional relations between the current object and its context. Then we combine the contextual information from the landmark feature convolution module with the target's visual features for grounding. To make this landmark feature convolution light-weight, we introduce a dynamic programming algorithm (termed dynamic max pooling) with low complexity to extract the landmark feature. Thanks to the landmark feature convolution module, we mimic the human behavior of 'Look Before You Leap' to design an LBYL-Net, which takes full consideration of contextual information. Extensive experiments show our method's effectiveness in four grounding datasets. Specifically, our LBYL-Net outperforms all state-of-the-art two-stage and one-stage methods on ReferitGame. On RefCOCO and RefCOCO+, Our LBYL-Net also achieves comparable results or even better results than existing one-stage methods. Code is available at https://github.com/svip-lab/LBYLNet.

show abstract